Kako masovno pingat?

ragezor je 24. apr 2015 ob 11:38 izjavil:

ker gradim nadzorni sistem

ragezor je 24. apr 2015 ob 11:33 izjavil:

niti ne vem tocno kje je problem. a je problem na ravni OSa, ker pac pingas full naenkrat. ali je problem v python threadih? glede na to, da ne pinga python proces se nagibam k temu, da je kriv OS.

ragezor je 24. apr 2015 ob 19:26 izjavil:

Zato pa je tema tule, da lahko predlagas boljse toole in arhitekture.

ragezor je 24. apr 2015 ob 20:40 izjavil:

Za IO bound aplikacije je ustvarjanje procesov slaba izbira, ker sele pri procesih pa res dobis dosti overheada

ragezor je 24. apr 2015 ob 20:40 izjavil:

+ ce procesi komunicirajo med sabo, morajo serializirati podatke, jih poslati in spet deserializirati. Pri threadih tega ni potrebno.

ragezor je 24. apr 2015 ob 20:40 izjavil:

AndrejO, jezik bi se ze naucil, da bi napisal en mikro servis, ki bi pingal, ce bi v kakem jeziku obstajala kaksna dobra knjiznica posebej za to. Da bi pa sel pisat specializiran ping v Cju (gradnja paketkov, crcjanje, itd,..) pa glih nebi.

ragezor je 24. apr 2015 ob 20:40 izjavil:

Cimvec seveda.
Cim manjse seveda :)

ragezor je 24. apr 2015 ob 20:40 izjavil:

terryww, ni vse kar najdes na stackoverflowu dobra koda. Za IO bound aplikacije je ustvarjanje procesov slaba izbira, ker sele pri procesih pa res dobis dosti overheada + ce procesi komunicirajo med sabo, morajo serializirati podatke, jih poslati in spet deserializirati. Pri threadih tega ni potrebno.

s primerom iz stackoverflowa sem ti želel nakazat kako, če že s threadi, ker ti zaženeš thread in v njem nov proces, skratka, kr nekaj

ker so threadi v pythonu potratni, sploh 1k threadov, kot je AndrejO že večkrat opozoril, sem ti nakazal drugo alternativo - korutine (v pythonu maš greenlet lib ipd.), ki je ravno mišljen za take probleme

za kaj bolj naprednega maš masscan in rešiš vse probleme

ragezor je 24. apr 2015 ob 21:13 izjavil:

Moja resitev ustvarja procese, ker je to bil najhitrejsi nacin implementacije in testiranje zadeve. Tudi python sem izbral zaradi tega, ker ga najbolj poznam in najlazje eksperimentiram.

ragezor je 24. apr 2015 ob 21:13 izjavil:

GIL ne vpliva tukaj dosti na zadevo.

ragezor je 24. apr 2015 ob 21:13 izjavil:

Najprej se delas norca, ker se mi zdi lahko nauciti en jezik za neko malo stvar, ki bi uporabljala neko knjiznico,

ragezor je 24. apr 2015 ob 21:13 izjavil:

kasneje se ti zdi cudno, da sem izbral python.

ragezor je 24. apr 2015 ob 21:13 izjavil:

A ti se za vsako stvar najprej pozanimas, kje se najhitreje naredi in se potem gres naucit jezik in potem se lotis zadeve? Se dobro, da projekti vsebujejo vedno samo en tip problemov, da lahko izberes potem jezik, ki je najboljsi za ta tip :)

ragezor je 24. apr 2015 ob 21:13 izjavil:

Ja, osnovno uporabo jezika (in knjiznice) se je verjento hitreje nauciti, kot prestudirati par protokolov, razumeti algoritme in potem implementirati zadevo, ki bo performirala boljse kot zadeva, ki so jo verjetno napisali eksperti na podrocju.

ragezor je 24. apr 2015 ob 21:13 izjavil:

Se vedno nobene pametne ideje ali namiga od tebe.

ragezor je 25. apr 2015 ob 00:03 izjavil:

ja, python. PC racunalnik oziroma povprecen server.

ragezor je 25. apr 2015 ob 00:03 izjavil:

Odstopanje mora biti manjse kot pri zdajsnjem pristopu. Nevem postaviti neke meje pri takih problemih, ker nikoli ne bo perfektno, ampak je pa fajn, da je cimboljse.

ragezor je 25. apr 2015 ob 00:03 izjavil:

medtem casom sem ze raziskoval in zgleda je point v tem, da je tvoj program kar se da odziven na select()

ragezor je 25. apr 2015 ob 00:03 izjavil:

bom jutri probal taktiko: 2 threada, en najprej ustvari vse paketke, potem jih pa zaporedoma poslje. drug thread pa v while zanki caka na select in shranjuje rezultate in timestampe nekam. ko je vsega konec se pa poracunajo zadeve.

ragezor je 25. apr 2015 ob 00:03 izjavil:

edino ni mi jasno, ce lahko jaz posljem tri paketke do enega hosta naenkrat (z razlicnimi sequenci), ali moram cakati, da dobim prvi paketek nazaj in sele nato lahko naslednjega posljem. pomoje, da lahko vse naenkrat. ne vidim razloga zakaj nebi.

pa kako vem, koliko byteov moram sprejemat. ker se bo nagrmadilo verjento vec paketkov zaporedoma bi jih rad jemal enega po enega.

ragezor je 28. apr 2015 ob 14:14 izjavil:

fajn bi bila moznost pinganja istega hosta naenkrat

import os
import select
import socket
import struct
import time


class Error(Exception):
  """Base error class for all ICMP exceptions."""
  def __init__(self, msg="base ping error"):
    super(Exception, self).__init__(msg)


class PingTimeout(Error):
  """Timeout waiting for ICMP Echo reply from destination."""
  def __init__(self, msg="ping timeout"):
    super(PingTimeout, self).__init__(msg)


class InvalidChecksum(Error):
  """Invalid ICMP packet checksum."""
  def __init__(self, msg="invalid checksum"):
    super(InvalidChecksum, self).__init__(msg)


class _ICMPEcho(object):
  """Encapsulation of ICMP Echo and Reply packets."""
  def __init__(self, type, code, id, seq, data):
    self.type = type
    self.code = code
    self.id = id
    self.seq = seq
    self.data = data

  def pack(self):
    fmt = "!BBHHH%ds" % len(self.data)
    pkt = struct.pack(fmt, self.type, self.code, 0, self.id, self.seq, self.data)
    csum = _checksum(pkt)
    return struct.pack(fmt, self.type, self.code, csum, self.id, self.seq, self.data)

  @staticmethod
  def unpack(pkt, has_ip_hdr=False):
    if has_ip_hdr:
      hdr_size = (ord(pkt[0]) & 0xf) << 2
      pkt = pkt[hdr_size:]
    (type, code, csum, id, seq, data) = struct.unpack("!BBHHH%ds" % (len(pkt) - 8), pkt)
    pktchk = struct.pack("!BBHHH%ds" % len(data), type, code, 0, id, seq, data)
    if csum != _checksum(pktchk):
      raise InvalidChecksum()
    return _ICMPEcho(type, code, id, seq, data)


def _checksum(buf):
  """Calculate one's complement of one's complement of sum of 16-bit words."""

  # If buffer length is odd, add 0 byte to it to even it out.
  if len(buf) & 1 == 1:
    buf = buf[:] + "\0"

  # Checksum loop calculates one's complement sum. 16-bit word is assumed
  # to be in network order.
  s = 0
  for ii in range(0, len(buf), 2):
    # Get 16-bit word, network order.
    w = ord(buf[ii + 1]) + (ord(buf[ii]) << 8)
    # Add with carry.
    s += w
    s = (s & 0xffff) + (s >> 16)

  # Checksum is one's complement of one's complement sum.
  return s ^ 0xffff


def ping(address, seq, timeout):
  """Send ICMP Echo request packet to address and wait for reply.

  Sends ICMP Echo request packet to address and waits for reply or timeout.

  Args:
    address: a string, address of host to ping.
    seq:     an int, sequence number between 1 and 65535.
    timeout: a float, time to wait for echo reply reply in seconds.
  Returns:
    a float, time in seconds that it took to receive echo reply.
  Raises:
    PingTimeout: when echo reply isn't received within specified timeout.
  """

  # Prepare ICMP Echo request packet. Discriminator is needed to avoid hitting
  # delayed packets in the same process, because processes could be reused.
  id = os.getpid() & 0xffff
  disc = "%f" % time.time()
  pkt = _ICMPEcho(8, 0, id, seq, disc)

  # Open socket and send prepared packet to destination address.
  s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_ICMP)
  s.sendto(pkt.pack(), (address, 0))

  # Store the time when packet was delivered to the system (assumed sent).
  start = time.time()

  # Deadline is the value of time.time() until which to wait for matching
  # ICMP Reply packet to return. It's used to calculate select() timeout.
  deadline = start + timeout

  while timeout > 0.0:
    # Wait for an event on socket.
    ready = select.select([s], [], [], timeout)

    # If there's nothing to read, a timeout occured.
    if len(ready[0]) == 0:
      raise PingTimeout()

    # Store received time. If packet will match, this will be used to calculate RTT.
    received = time.time()
    pkt, src_addr = s.recvfrom(1528)

    # Short the loop if the packet is from the wrong address.
    if src_addr[0] != address:
      timeout = deadline - time.time()
      continue

    # Short the loop if packet doesn't match the one we have sent.
    pkt = _ICMPEcho.unpack(pkt, has_ip_hdr=True)
    if pkt.type != 0 or pkt.code != 0 or pkt.id != id or pkt.seq != seq or pkt.data != disc:
      timeout = deadline - time.time()
      continue

    # Calculate RTT and return it
    return received - start

import sys
import time

from multiprocessing import Pool

import ping

 
class Ping(object):
  def __init__(self, tries=3, interval=1.0, timeout=1.0):
    self.tries = tries
    self.interval = interval
    self.timeout = timeout

  def __call__(self, address):
    times = []
    for seq in xrange(1, self.tries + 1):
      not_before = time.time() + self.interval
      try:
        times.append(ping.ping(address, seq, self.timeout))
      except ping.PingTimeout:
        times.append(None)
      except Exception as e:
        return (address, e)
      # Wait if needed before moving to the next iteration.
      while seq < self.tries and time.time() < not_before:
        time.sleep(not_before - time.time())
    # Return address and list of tries.
    return (address, times)


def ping_all(addresses, processes=25):
  pool = Pool(processes=processes)
  ok, err = [], []
  for res in pool.imap(Ping(), addresses, chunksize=10):
    if isinstance(res[1], Exception):
      err.append(res)
    elif not [x for x in res[1] if x]:
      err.append((res[0], Exception("host unreachable")))
    else:
      ok.append(res)
  return ok, err


def main(argv):
  addresses = ['127.0.0.1'] * 1000

  start = time.time()
  ok, err = ping_all(addresses)
  end = time.time()

  if ok:
    print "Averages:"
    for result in ok:
      valid = [rtt for rtt in result[1] if rtt]
      success = len(valid) * 100 / len(result[1])
      print "  %s:\t%%/Avg/Min/Max: %3d%%/%f/%f/%f" % (result[0], success, sum(valid)/float(len(valid)), min(valid), max(valid))

  if err:
    print "Errors:"
    for result in err:
      print "  %s:\t%s" % result

  print "Time to execute all: %f" % (end - start)


if __name__ == "__main__":
    sys.exit(main(sys.argv))

ragezor je 28. apr 2015 ob 18:25 izjavil:

lahko prosim se malo razlozis kaj se dogaja v pack() in unpack()

ragezor je 28. apr 2015 ob 18:25 izjavil:

ter kaj je type, code in id?