GW labs

Docker Machine with USB support on Windows/macOS

2017-02-07T00:00:00+01:00

Docker Machine with USB support on Windows/macOS

The only way to make USB devices available in Docker containers under Windows or macOS is to use the Docker Machine that comes with the usual installer and uses VirtualBox. This is not needed on a Linux host OS with x86_64 architecture.

Download and install Docker for Windows or Docker for Mac (Docker Machine) from https://www.docker.com/products/docker.
Download and install VirtualBox 5.1+ from https://www.virtualbox.org/wiki/Downloads.
Optionally, to make flashing faster also download and install VirtualBox Extension Pack.
Initialize the Docker Machine as a virtual machine (VM) called default from the console:

$ docker-machine create --driver virtualbox default
Running pre-create checks...
(default) No default Boot2Docker ISO found locally, downloading the latest release...
(default) Latest release for github.com/boot2docker/boot2docker is v1.13.0
(default) Downloading /Users/ami/.docker/machine/cache/boot2docker.iso from https://github.com/boot2docker/boot2docker/releases/download/v1.13.0/boot2docker.iso...
(default) 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
Creating machine...
(default) Copying /Users/ami/.docker/machine/cache/boot2docker.iso to /Users/ami/.docker/machine/machines/default/boot2docker.iso...
(default) Creating VirtualBox VM...
(default) Creating SSH key...
(default) Starting the VM...
(default) Check network to re-create if needed...
(default) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env default
$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
default   -        virtualbox   Running   tcp://192.168.99.100:2376           v1.13.0

Enable USB support in VirtualBox for your VM default:

Note: If you installed the VirtualBox Extension Pack, apply the --usbxhci on option instead of --usb on .

$ docker-machine stop
Stopping "default"...
Machine "default" was stopped.
$ vboxmanage modifyvm default --usb on
$ #vboxmanage modifyvm default --usbxhci on
$ docker-machine start
Starting "default"...
(default) Check network to re-create if needed...
(default) Waiting for an IP...
Machine "default" was started.
Waiting for SSH to be available...
Detecting the provisioner...
Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.

One way to attach USB devices in your VM default is to add filter rules that automatically attach USB devices:

Note: First command determines which USB devices are available on your host, following commands add filter rules that automatically attach USB devices with a matching vendor id and product id. Afterwards you will need to unplug and plug those USB devices back in (or restart your VM), to trigger the attach filters.

$ vboxmanage list usbhost
Host USB Devices:

UUID:               5b23cdc4-cfc7-49c4-81f6-844b1378cdf1
VendorId:           0x0403 (0403)
ProductId:          0x6001 (6001)
Revision:           6.0 (0600)
Port:               2
USB version/speed:  0/Full
Manufacturer:       FTDI
Product:            TTL232R-3V3
SerialNumber:       FTGI13S5
Address:            p=0x6001;v=0x0403;s=0x0000f62c7b7304f3;l=0x14212000
Current State:      Busy

$ vboxmanage usbfilter add 0 --target default --name 'FTDI TTL232R-3V3' --vendorid 0x0403 --productid 0x6001

Alternatively, you can attach USB devices manually:
- open the VirtualBox application
- find your VM default
- select Show or double-click on it
- wait for the VM window to show up
- select Devices/USB from the menu and enable what devices you want to use from the host in the guest (and consequently in Docker containers)
If you want to tweak something in your VM default itself, you can SSH into it with:

$ docker-machine ssh

Activate correct environment to use the Docker Machine in VM default:

Warning: Do not forget to run this each time you open a new console and want to begin using Docker!!!

$ eval "$(docker-machine env default)"

Now you are ready to use Docker.

Docker dovecot-getmail

2016-12-16T00:00:00+01:00

docker-dovecot-getmail is a Docker image based on Debian 8 implementing a private email gateway with dovecot and getmail for gathering emails from multiple accounts on a private server (IMAP), but using a public email infrastructure for sending (SMTP).

It is a Docker container realizing a similar architecture to:

http://joel.porquet.org/wiki/hacking/getmail_dovecot/

+-----------+              +-----------+               +--------------+
| ISP       |              | DOCKER    |               | LAPTOP       |
|           |              |           |           +-->|--------------|
| +-------+ | push/delete  | +-------+ | push/sync |   |  MAIL CLIENT +---+
| | IMAPS +----------------->| IMAPS +<------------+   +--------------+   |
| +-------+ |              | +-------+ |           |   +--------------+   |
| +-------+ |              |           |           |   | ANDROID      |   |
| | SMTP  |<-------+       |           |           +-->|--------------|   |
| +-------+ |      |       |           |               |  MAIL CLIENT +---+
+-----------+      |       +-----------+               +--------------+   |
                   +------------------------------------------------------+

Open source project:

home: http://gw.tnode.com/docker/dovecot-getmail/
github: http://github.com/gw0/docker-dovecot-getmail/
technology: debian, dovecot, getmail
docker hub: https://hub.docker.com/r/gw000/dovecot-getmail/

Usage

Requirements:

/home: mounted users directories (Maildir in fs layout, sieve, .getmail)
/etc/cron.d: mounted crontabs for executing all getmail accounts
/etc/ssl/private: mounted SSL/TLS certificates (dovecot.crt, dovecot.key)

Prepare your getmailrc account configurations per user (/srv/mail/home/user/.getmail/getmailrc-user@email.invalid):

# ~/.getmail/getmailrc-*: getmailrc email configuration

[retriever]
type = SimpleIMAPSSLRetriever
server = imap.email.invalid
username = user@email.invalid
port = 993
password = password
mailboxes = ("INBOX", "Sent", "Spam")

[destination]
type = MDA_external
path = /usr/lib/dovecot/deliver
arguments = ("-e",)

[options]
read_all = false
delete_after = 30
delivered_to = false
received = true
verbose = 1

If you are using Sieve filters and want a Refilter mailbox to trigger their refiltering, create a refilter configuration per user (/srv/mail/home/user/.getmail/getmailrc-refilter):

# ~/.getmail/getmailrc-*: getmailrc refilter configuration

[retriever]
type = SimpleIMAPRetriever
server = localhost
port = 143
username = user
password = password
mailboxes = ("Refilter",)

[destination]
type = MDA_external
path = /usr/lib/dovecot/deliver
arguments = ("-e",)

[options]
read_all = false
delete = true
delivered_to = false
received = false
verbose = 1

Prepare crontab file (/srv/mail/cron.d/getmail) for periodically checking for new mail for each user and account:

# /etc/cron.d/getmail: system-wide crontab for getmail
SHELL=/bin/sh

# m h dom mon dow user  command
*/20 *  *   *   * user  ACC="user-refilter" && (date; flock -n ~/.getmail/lock-$ACC getmail --rcfile="getmailrc-$ACC" --idle Refilter) >>"/var/log/getmail/$ACC.log" 2>&1
*/20 *  *   *   * user  ACC="user@email.invalid" && (date; flock -n ~/.getmail/lock-$ACC getmail --rcfile="getmailrc-$ACC" --idle INBOX) >>"/var/log/getmail/$ACC.log" 2>&1

Do not forget to place your SSL certificates as /srv/mail/ssl/dovecot.crt and /srv/mail/ssl/dovecot.key. SSL is required!

And finally start it with docker:

$ docker run -d -v /srv/mail/home:/home -v /srv/mail/cron.d:/etc/cron.d -v /srv/mail/ssl:/etc/ssl/private:ro -p 143 -p 993 -p 4190 --name mail gw000/dovecot-getmail

Or use docker-compose (check out docker-compose.example.yml).

Users are created automatically with default password (replaceMeNow) on first start. To reset user passwords (of a running container):

$ docker exec -it mail passwd user

Feedback

If you encounter any bugs or have feature requests, please file them in the issue tracker or even develop it yourself and submit a pull request over GitHub.

License

This library is licensed under the GNU Affero General Public License 3.0+ (AGPL-3.0+). Note that it is mandatory to make all modifications and complete source code of this library publicly available to any user.

Discourse Sense Classification from Scratch using Focused RNNs

2016-08-12T00:00:00+02:00

Micro Drone 3.0 Camera API

2016-07-30T00:00:00+02:00

Micro Drone 3.0 is an affordable feature-packed mini quadcopter. It can be controlled either using the 2.4 GHz remote controller (receiver on main board) or an Android/iPhone app over WiFi (through the camera module). Although the project was crowdfunded on Indiegogo, the software is not open source and no API documentation is available. This is an attempt to reverse engineer the main parts of the Camera API, so anyone can experiment further.

General

The Micro Drone 3.0 Camera module takes around 20 seconds to boot when you attach it to the battery. It creates an WiFi access point (eg. MD3.0_ABCD) to which you need to connect your phone before starting the Micro Drone app.

The Camera module is on 192.168.1.1 and provides the following services:

67/udp: DHCP daemon (udhcpd from BusyBox v1.16.1)
80/tcp: HTTP server and controller (/bin/camera using Boa/0.94.14rc21)
2345/tcp: unknown (/bin/camera)
10000/udp: unknown (/bin/camera)
23/tcp: telnet daemon with root shell (telnetd) (needs to be activated)

The Android/iPhone app communicates for camera operations, video streaming and serial remote control only over port 80/tcp to the main daemon (/bin/camera). On the same port this daemon exposes an HTTP interface, but also a custom protocol for streaming video and serial remote control commands.

HTTP API

Over HTTP a Reecam CGI interface is available (user admin without any password), but it is easier to use the handy mdcam tool to work with the camera.

To download all photos and videos to your computer with the mdcam tool:

$ mdcam ls | cut -d' ' -f1 | xargs -n 1 mdcam download

The full HTTP API is:

/get_status.cgi
/get_params.cgi
/get_properties.cgi
/check_user.cgi
/get_badauth.cgi
/create_session.cgi
/close_session.cgi
/is_mjpeg_stream_exist.cgi
/request_av.cgi
/set_params.cgi
/snapshot.cgi
/videostream.cgi
/ptz_control.cgi
/get_log.cgi
/wifi_scan.cgi
/get_session_list.cgi
/search_record.cgi
/del_record.cgi
/get_record.cgi
/start_record.cgi
/stop_record.cgi
/format_sd.cgi
/backup.cgi
/restore.cgi
/restore_factory.cgi
/delete_factory.cgi
/upgrade.cgi
/write_comm.cgi
/read_comm.cgi
/test_wifi_connected.cgi
/alarm_snapshots.cgi
/search_snapshot.cgi
/del_snapshot.cgi
/get_snapshot.cgi
/set_stream.cgi
/get_cur_ir_adc_value.cgi
/wifi_ate_tx_start.cgi

To get a live video stream open the following network URL in VLC media player:

http://192.168.1.1/av.asf

Telnet shell

To activate the Telnet interface enable it though the HTTP API by visiting:

http://192.168.1.1/set_params.cgi?telnetd=1&save=1&reboot=1

After a reboot you can connect over telnet into a root shell with BusyBox v1.16.1:

$ telnet 192.168.1.1
Trying 192.168.1.1...
Connected to 192.168.1.1.
Escape character is '^]'.

# uname -a
Linux (none) 3.0.8 #1723 Wed Nov 25 16:49:38 CST 2015 armv5tejl GNU/Linux
# cat /proc/mtd 
dev:    size   erasesize  name
mtd0: 00100000 00010000 "boot"
mtd1: 00500000 00010000 "kernel"
mtd2: 00080000 00010000 "user"
mtd3: 00180000 00010000 "manufacturer"
# df
Filesystem           1K-blocks      Used Available Use% Mounted on
tmpfs                    18056         4     18052   0% /dev
/dev/mtdblock2             512       252       260  49% /mnt/user
/dev/mtdblock3            1536       220      1316  14% /mnt/manufacturer
/dev/mmcblk0p1        15621208   1151032  14470176   7% /mnt/sd
# mount
rootfs on / type rootfs (rw)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
tmpfs on /dev type tmpfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
/dev/mtdblock2 on /mnt/user type jffs2 (rw,relatime)
/dev/mtdblock3 on /mnt/manufacturer type jffs2 (rw,relatime)
/dev/mmcblk0p1 on /mnt/sd type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

It is a little tricky to retrieve the /bin/camera binary, because it removes itself after starting. Nevertheless it is still loaded in the memory, so you can find it in /proc/*/exe. It seems that they expected someone to examine this binary, because it contains a string hello kitty and kgb/cia 2011 COPYRIGHT@REECAM 5460.

Serial interface

The Android/iPhone app also manages to send remote control commands the drone. This is done through a serial interface that seems to be directly exposed to the app through libcamlib.so on the client side and /bin/camera on the server side.

Disassembling the Android APK reveals that com.D_Lawliet.fly.FlyActivity contains the targets = new byte[7] property that encodes the serial commands that are being sent with each packet. Meaning of those bytes:

targets[0]: -6
targets[1]: throttle (0 is off)
targets[2]: rudder (min 1?)
targets[3]: elevation (1..127)
targets[4]: aileron (1..127)
targets[5]: 0 + settings (speed mode, inverted flying)
targets[6]: checksum ([1] xor [2] xor [3] xor [4] xor[5]`)

The target serial device for this commands is /dev/ttyAMA1. Proof of concept that triggers throttle for a few seconds:

# echo -e "\xfa\x40\x40\x40\x40\x00\x00" > /dev/ttyAMA1

Discourse Sense Classification from Scratch using Focused RNNs

2016-08-12T00:00:00+02:00

Conference proceeding

G. Weiss and M. Bajec, “Discourse Sense Classification from Scratch using Focused RNNs,” in Proceedings of the Twentieth Conference on Computational Natural Language Learning - Shared Task, 2016, pp. 50–54.

Abstract

The subtask of CoNLL 2016 Shared Task focuses on sense classification of multilingual shallow discourse relations. Existing systems rely heavily on external resources, hand-engineered features, patterns, and complex pipelines fine-tuned for the English language. In this paper we describe a different approach and system inspired by end-to-end training of deep neural networks. Its input consists of only sequences of tokens, which are processed by our novel focused RNNs layer, and followed by a dense neural network for classification. Neural networks implicitly learn latent features useful for discourse relation sense classification, make the approach almost language-agnostic and independent of prior linguistic knowledge. In the closed-track sense classification our system achieved overall 0.5246 F1-measure on English blind dataset and achieved the new state-of-the-art of 0.7292 F1-measure on Chinese blind dataset.

Docker debian-cuda

2016-12-21T00:00:00+01:00

docker-debian-cuda is a minimal Docker image built from Debian 9 (amd64) with CUDA Toolkit and cuDNN using only Debian packages.

Although the nvidia-docker tool can run CUDA inside Docker images, it uses yet another wrapper command and is based on Ubuntu images. To make the whole process more transparent, we explicitly expose GPU devices and build from official Debian images. All installations are performed through the Debian package manager, also because the official Nvidia CUDA Toolkit does not support Debian without hacks.

Open source project:

home: http://gw.tnode.com/docker/debian-cuda/
github: http://github.com/gw0/docker-debian-cuda/
technology: debian, cuda toolkit, opencl
docker hub: http://hub.docker.com/r/gw000/debian-cuda/

Available tags:

8.0.44-2_5.1.5-1_375.20-4, 8.0_5.1, latest [2016-12-21]: CUDA Toolkit (8.0.44-2) + cuDNN (5.1.5-1) + CUDA library (375.20-4) (Dockerfile)
7.5.18-4_5.1.3_361.45.18-2, 7.5_5.1 [2016-09-19]: CUDA Toolkit (7.5.18-4) + cuDNN (5.1.3) + CUDA library (361.45.18-2) (Dockerfile)
7.5.18-2 [2016-07-20]: CUDA Toolkit (7.5.18-2) + cuDNN (4.0.7) + CUDA library (352.79-8)

Usage

Host system requirements (eg. Debian 8 or 9):

CUDA-capable GPU
nvidia-kernel-dkms (same as CUDA library)
optionally nvidia-cuda-mps, nvidia-smi

To utilize your GPUs this Docker image needs access to your /dev/nvidia* devices, like:

$ docker run -it --rm $(ls /dev/nvidia* | xargs -I{} echo '--device={}') gw000/debian-cuda

Host system

List of devices that should be present on the host system:

$ ll /dev/nvidia*
crw-rw---- 1 root video 250,   0 Jul 13 15:56 /dev/nvidia-uvm
crw-rw---- 1 root video 250,   1 Jul 13 15:56 /dev/nvidia-uvm-tools
crw-rw---- 1 root video 195,   0 Jul 13 15:56 /dev/nvidia0
crw-rw---- 1 root video 195, 255 Jul 13 15:56 /dev/nvidiactl

In case /dev/nvidia0 and /dev/nvidiactl are not present, ensure the kernel module nvidia is automatically loaded, properly configured and optimized, and there is a udev rule to create the devices:

$ echo 'nvidia' > /etc/modules-load.d/nvidia.conf
$ cat > /etc/udev/rules.d/70-nvidia.rules << __EOF__
KERNEL=="nvidia", RUN+="/bin/bash -c '/usr/bin/nvidia-smi -L && /bin/chmod 0660 /dev/nvidia* && /bin/chgrp video /dev/nvidia*'"
__EOF__

For OpenCL support the devices /dev/nvidia-uvm and /dev/nvidia-uvm-tools are needed. Ensure the kernel module nvidia-uvm is automatically loaded, and add a custom udev rule to create the device:

$ echo 'nvidia-uvm' > /etc/modules-load.d/nvidia-uvm.conf
$ cat > /etc/udev/rules.d/70-nvidia-uvm.rules << __EOF__
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -u && /bin/chmod 0660 /dev/nvidia-uvm* && /bin/chgrp video /dev/nvidia-uvm*'"
__EOF__

If you would like to monitor real-time temperatures on your host system use something like:

$ watch -n 5 'nvidia-smi; echo; sensors; for hdd in /dev/sd?; do echo -n "$hdd  "; smartctl -A $hdd | grep Temperature_Celsius; done'

In case your Nvidia kernel driver and CUDA library versions differ an error appears in kernel messages (dmesg) or using nvidia-smi inside the container. Possible solutions:

upgrade your Nvidia kernel driver on the host directly from Debian 9 packages: nvidia-kernel-dkms, nvidia-alternative, libnvidia-ml1, nvidia-smi
upgrade your Nvidia kernel driver on the host by compiling it yourself
inject the correct version of CUDA library into the container (if it is installed on the host) with:

$ docker run -it --rm $(ls /dev/nvidia* | xargs -I{} echo '--device={}') $(ls /usr/lib/x86_64-linux-gnu/libcuda.* | xargs -I{} echo '-v {}:{}:ro') gw000/debian-cuda

Feedback

If you encounter any bugs or have feature requests, please file them in the issue tracker or even develop it yourself and submit a pull request over GitHub.

License

Docker keras-full

2017-01-19T00:00:00+01:00

docker-keras-full is a Docker image built from Debian 9 (amd64) with a full reproducible deep learning research environment based on Keras and Jupyter. It supports CPU and GPU processing with Theano and TensorFlow backends. It features Jupyter Notebook with Python 2 and 3 support and uses only Debian and Python packages (no manual installations).

Open source project:

home: http://gw.tnode.com/docker/keras-full/
github: http://github.com/gw0/docker-keras-full/
technology: debian, keras, theano, tensorflow, openblas, cuda toolkit, python, numpy, h5py, jupyter, matplotlib, pillow, pandas, scikit-learn, statsmodels
docker hub: http://hub.docker.com/r/gw000/keras-full/

Available tags:

1.2.0, latest [2016-12-21]: Python 2.7/3.5 + Keras (1.2.0) + TensorFlow (0.12.0) + Theano (0.8.2) on CPU/GPU
1.1.0 [2016-09-20]: Python 2.7/3.5 + Keras (1.1.0) + TensorFlow (0.10.0) + Theano (0.8.2) on CPU/GPU
1.0.8 [2016-08-28]: Python 2.7/3.5 + Keras (1.0.8) + TensorFlow (0.9.0) + Theano (0.8.2) on CPU/GPU
1.0.6 [2016-07-20]: Python 2.7/3.5 + Keras (1.0.6) + TensorFlow (0.9.0) + Theano (0.8.2) on CPU/GPU
1.0.4 [2016-06-16]: Python 2.7/3.5 + Keras (1.0.4) + TensorFlow (0.8.0) + Theano (0.8.2) on CPU/GPU

Usage

Quick experiment from console with IPython 2.7 or 3.5:

$ docker run -it --rm gw000/keras-full ipython2
$ docker run -it --rm gw000/keras-full ipython3

To start the Jupyter IPython web interface on http://<ip>:8888/ (password: keras) and notebooks stored in /srv/notebooks:

$ docker run -d -p=6006:6006 -p=8888:8888 -v=/srv/notebooks:/srv gw000/keras-full

To utilize your GPUs this Docker image needs access to your /dev/nvidia* devices (see docker-debian-cuda), like:

$ docker run -d $(ls /dev/nvidia* | xargs -I{} echo '--device={}') -p=6006:6006 -p=8888:8888 -v=/srv/notebooks:/srv gw000/keras-full

To change the default password, prepare a new hashed password and pass it as an environment variable:

$ docker run -d -p=6006:6006 -p=8888:8888 -e PASSWD="sha1:..." -v=/srv/notebooks:/srv gw000/keras-full

Feedback

If you encounter any bugs or have feature requests, please file them in the issue tracker or even develop it yourself and submit a pull request over GitHub.

License

Docker keras

2017-01-26T00:00:00+01:00

docker-keras is a minimal Docker image built from Debian 9 (amd64) for reproducible deep learning based on Keras. It features minimal images for Python 2 or 3, TensorFlow or Theano backends, processing on CPU or GPU, and uses only Debian and Python packages (no manual installations).

Open source project:

home: http://gw.tnode.com/docker/keras/
github: http://github.com/gw0/docker-keras/
technology: debian, keras, tensorflow, theano, openblas, cuda toolkit, python, numpy, h5py
docker hub: https://hub.docker.com/r/gw000/keras/

Available tags:

1.2.1-py2, 1.2.1-cpu, 1.2.1, latest points to 1.2.1-py2-tf-cpu
1.2.1-py3 points to 1.2.1-py3-tf-cpu
1.2.1-gpu points to 1.2.1-py2-tf-gpu
1.2.1-py2-tf-cpu/1.2.1-py2-tf-gpu [2017-01-20]: Python 2.7 + Keras (1.2.1) + TensorFlow (0.12.1) on CPU/GPU (Dockerfile.py2-tf-cpu/.py2-tf-gpu)
1.2.1-py2-th-cpu/1.2.1-py2-th-gpu [2017-01-20]: Python 2.7 + Keras (1.2.1) + Theano (0.8.2) on CPU/GPU (Dockerfile.py2-th-cpu/.py2-th-gpu)
1.2.1-py3-tf-cpu/1.2.1-py3-tf-gpu [2017-01-20]: Python 3.5 + Keras (1.2.1) + TensorFlow (0.12.1) on CPU/GPU (Dockerfile.py3-tf-cpu/.py3-tf-gpu)
1.2.1-py3-th-cpu/1.2.1-py3-th-gpu [2017-01-20]: Python 3.5 + Keras (1.2.1) + Theano (0.8.2) on CPU/GPU (Dockerfile.py3-th-cpu/.py3-th-gpu)
1.2.0-py2-tf-cpu/1.2.0-py2-tf-gpu [2016-12-20]: Python 2.7 + Keras (1.2.0) + TensorFlow (0.12.0) on CPU/GPU
1.2.0-py2-th-cpu/1.2.0-py2-th-gpu [2016-12-20]: Python 2.7 + Keras (1.2.0) + Theano (0.8.2) on CPU/GPU
1.2.0-py3-tf-cpu/1.2.0-py3-tf-gpu [2016-12-20]: Python 3.5 + Keras (1.2.0) + TensorFlow (0.12.0) on CPU/GPU
1.2.0-py3-th-cpu/1.2.0-py3-th-gpu [2016-12-20]: Python 3.5 + Keras (1.2.0) + Theano (0.8.2) on CPU/GPU
1.1.1-py2-tf-cpu/1.1.1-py2-tf-gpu [2016-10-31]: Python 2.7 + Keras (1.1.1) + TensorFlow (0.10.0) on CPU/GPU
1.1.1-py2-th-cpu/1.1.1-py2-th-gpu [2016-10-31]: Python 2.7 + Keras (1.1.1) + Theano (0.8.2) on CPU/GPU
1.1.1-py3-tf-cpu/1.1.1-py3-tf-gpu [2016-10-31]: Python 3.5 + Keras (1.1.1) + TensorFlow (0.10.0) on CPU/GPU
1.1.1-py3-th-cpu/1.1.1-py3-th-gpu [2016-10-31]: Python 3.5 + Keras (1.1.1) + Theano (0.8.2) on CPU/GPU
1.1.0-py2-tf-cpu/1.1.0-py2-tf-gpu [2016-09-20]: Python 2.7 + Keras (1.1.0) + TensorFlow (0.10.0) on CPU/GPU
1.1.0-py2-th-cpu/1.1.0-py2-th-gpu [2016-09-20]: Python 2.7 + Keras (1.1.0) + Theano (0.8.2) on CPU/GPU
1.1.0-py3-tf-cpu/1.1.0-py3-tf-gpu [2016-09-20]: Python 3.5 + Keras (1.1.0) + TensorFlow (0.10.0) on CPU/GPU
1.1.0-py3-th-cpu/1.1.0-py3-th-gpu [2016-09-20]: Python 3.5 + Keras (1.1.0) + Theano (0.8.2) on CPU/GPU
1.0.8-py2-tf-cpu/1.0.8-py2-tf-gpu [2016-08-28]: Python 2.7 + Keras (1.0.8) + TensorFlow (0.9.0) on CPU/GPU
1.0.8-py2-th-cpu/1.0.8-py2-th-gpu [2016-08-28]: Python 2.7 + Keras (1.0.8) + Theano (0.8.2) on CPU/GPU
1.0.8-py3-tf-cpu/1.0.8-py3-tf-gpu [2016-08-28]: Python 3.5 + Keras (1.0.8) + TensorFlow (0.9.0) on CPU/GPU
1.0.8-py3-th-cpu/1.0.8-py3-th-gpu [2016-08-28]: Python 3.5 + Keras (1.0.8) + Theano (0.8.2) on CPU/GPU
1.0.6-py2-tf-cpu/1.0.6-py2-tf-gpu [2016-07-20]: Python 2.7 + Keras (1.0.6) + TensorFlow (0.9.0) on CPU/GPU
1.0.6-py2-th-cpu/1.0.6-py2-th-gpu [2016-07-20]: Python 2.7 + Keras (1.0.6) + Theano (0.8.2) on CPU/GPU
1.0.6-py3-tf-cpu/1.0.6-py3-tf-gpu [2016-07-20]: Python 3.5 + Keras (1.0.6) + TensorFlow (0.9.0) on CPU/GPU
1.0.6-py3-th-cpu/1.0.6-py3-th-gpu [2016-07-20]: Python 3.5 + Keras (1.0.6) + Theano (0.8.2) on CPU/GPU
1.0.4-py2-tf-cpu/1.0.4-py2-tf-gpu [2016-06-16]: Python 2.7 + Keras (1.0.4) + TensorFlow (0.8.0) on CPU/GPU
1.0.4-py2-th-cpu/1.0.4-py2-th-gpu [2016-06-16]: Python 2.7 + Keras (1.0.4) + Theano (0.8.2) on CPU/GPU
1.0.4-py3-tf-cpu/1.0.4-py3-tf-gpu [2016-06-16]: Python 3.5 + Keras (1.0.4) + TensorFlow (0.8.0) on CPU/GPU
1.0.4-py3-th-cpu/1.0.4-py3-th-gpu [2016-06-16]: Python 3.5 + Keras (1.0.4) + Theano (0.8.2) on CPU/GPU
1.0.1-py2-th-cpu/1.0.1-py2-th-gpu [2016-04-16]: Python 2.7 + Keras (1.0.1) + Theano (0.8.1) on CPU/GPU
0.3.3-py2-th-cpu/0.3.3-py2-th-gpu [2016-03-31]: Python 2.7 + Keras (0.3.3) + Theano (0.8.1) on CPU/GPU

Usage

Quick experiment with latest Keras (with TensorFlow backend on CPU) and your Python 2 code in /srv/ai:

$ docker run -it --rm -v /srv/ai:/srv/ai gw000/keras /srv/ai/run.py

Or using TensorFlow backend on GPUs (see docker-debian-cuda) in Python 2:

$ docker run -it --rm $(ls /dev/nvidia* | xargs -I{} echo '--device={}') -v /srv/ai:/srv/ai gw000/keras:1.2.0-py2-tf-gpu /srv/ai/run.py

Or using Theano backend on GPUs (see docker-debian-cuda) in Python 3:

$ docker run -it --rm $(ls /dev/nvidia* | xargs -I{} echo '--device={}') -v /srv/ai:/srv/ai gw000/keras:1.2.0-py3-th-gpu /srv/ai/run.py

In practice you are supposed to extend this image by writing your own Dockerfile that installs all your application dependencies (either using apt-get or pip). Eg. if you need Matplotlib, PIL/pillow, Pandas, Scikit-learn, and Statsmodels:

FROM gw000/keras:1.2.0-py2-th-cpu

# install dependencies from debian packages
RUN apt-get update -qq \
 && apt-get install --no-install-recommends -y \
    python-matplotlib \
    python-pillow

# install dependencies from python packages
RUN pip --no-cache-dir install \
    pandas \
    scikit-learn \
    statsmodels

# install your app
ADD ai/ /srv/ai/
RUN chmod +x /srv/ai/run.py

CMD ["/srv/ai/run.py"]

If you are looking for a full deep learning research environment based on Keras and Jupyter, check out docker-keras-full.

Feedback

If you encounter any bugs or have feature requests, please file them in the issue tracker or even develop it yourself and submit a pull request over GitHub.

License

Docker periodic-sync

2016-12-02T00:00:00+01:00

docker-periodic-rsync is a Docker image based on Debian 8 with cron, ssh and rsync for periodic or one-time remote rsync copy jobs.

Open source project:

home: http://gw.tnode.com/docker/periodic-rsync/
github: http://github.com/gw0/docker-periodic-rsync/
technology: debian, cron, ssh, rsync
docker hub: https://hub.docker.com/r/gw000/periodic-rsync/

Usage

Requirements:

setup passwordless SSH login on remote machines (setup)
/root/.ssh: mount your passwordless SSH public and private keys (id_rsa/id_rsa.pub, chown to user root)
/data: mount preferred target directory
/etc/crontab: mount your crontab file (for periodic usage)

For one-time usage (need specify command):

$ docker run -it --rm -v /srv/backup/.ssh:/root/.ssh -v /srv/backup/data:/data gw000/periodic-rsync rsync -zave ssh user@server.remote:dir/ /data

For periodic usage (prepare crontab file /srv/backup/cron.d/backup):

# /etc/cron.d/backup: system-wide crontab
SHELL=/bin/sh

# m h dom mon dow user  command
*/5 *   *   *   * root  rsync -zave ssh user@server.remote:dir/ /data

$ docker run -d -v /srv/backup/.ssh:/root/.ssh -v /srv/backup/cron.d:/etc/cron.d -v /srv/backup/data:/data --name backup gw000/periodic-rsync

Feedback

If you encounter any bugs or have feature requests, please file them in the issue tracker or even develop it yourself and submit a pull request over GitHub.

License

Docker Compose organized naming convention

2016-03-10T00:00:00+01:00

Install Docker Compose

Docker Compose 1.6.2 does not have a Debian package, so instead of installing its Python dependencies on the host, it can be run on-demand inside a Docker container with a helper script.

$ wget -O /usr/local/bin/docker-compose https://github.com/docker/compose/releases/download/1.6.2/run.sh
$ chmod +x /usr/local/bin/docker-compose

Organized naming convention

Instead of having Docker containers inside /var/lib/docker, deployment repositories and data volumes all over the place, the following naming convention has proven useful:

/srv/docker/   -- Docker containers (from /var/lib/docker/)
/srv/repos/    -- per-project deployment repositories
/srv/repos/project1/docker-compose.yaml
/srv/repos/project1/docker-db/Dockerfile
/srv/repos/project1/docker-web/Dockerfile
/srv/storage/  -- per-container data volumes mounted on host
/srv/storage/project1/db
/srv/storage/project1/web

Initializing such a setup can be accomplished just by moving some files, making a symbolic link, and empty directories:

$ systemctl stop docker
$ mv /var/lib/docker/ /srv/docker/ && ln -s /srv/docker/ /var/lib/docker
$ mkdir /srv/repos/ /srv/storage/
$ systemctl start docker

Afterwards bringing up your infrastructure is simple as:

$ cd /srv/repos/project1
$ docker-compose up -d
$ docker-compose logs

Project development lifecycle

First prepare an empty Git repository on your Docker server:

$ NAME=project1
$ mkdir /srv/repos/$NAME && cd /srv/repos/$NAME
$ git init
$ git config receive.denycurrentbranch false
$ echo -e '#!/bin/sh\ngit --work-tree=.. checkout -f' > ./.git/hooks/post-receive && chmod +x ./.git/hooks/post-receive

Setup your project repository and code for deployment with Docker Compose on your workstation:

$ NAME=project1 SERVER=foo
$ mkdir ./$NAME && cd ./$NAME
$ git init
$ git remote add $SERVER ssh://$SERVER/srv/repos/own.tnode.com
$ git add *
$ git commit -m 'Initial commit for deploying.'
$ git push --set-upstream $SERVER master

Afterwards bringing up your infrastructure as described above:

$ cd /srv/repos/project1
$ docker-compose up -d
$ docker-compose logs

Docker installation on Debian 8

2016-03-10T00:00:00+01:00

Install Docker Engine

Docker Engine 1.10.2 requires a 64-bit OS with at least kernel version 3.10. Official Debian repositories are a few versions behind with Docker versions it is best to install it directly from Docker’s APT repository.

$ apt-get install linux-image-4.3.0-0.bpo.1-amd64
$ vi /etc/default/grub
GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"
$ update-grub
$ reboot

$ apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
$ echo 'deb http://apt.dockerproject.org/repo debian-jessie main' > /etc/apt/sources.list.d/docker.list
$ apt-get update
$ apt-get install docker-engine

Journald logging driver

Debian 8 switched to systemd as its default system and service manager, so it would make sense to also store log messages from containers in systemd journal (journald logging driver). Also note that Docker by default logs messages to a JSON file (json-file logging driver) that can get corrupted in some situations.

Logging driver can be configured by passing the --log-driver=journald option to the Docker daemon. Afterwards log messages can be retrieved with:

$ journalctl CONTAINER_NAME=web

Overlay storage driver

The default aufs storage driver is production-ready and stable, but it is not efficient under high write activity and not included in mainline Linux kernel. Recently overlay storage driver is gaining popularity and available in kernel since 3.18 (apt-get install linux-image-4.3.0-0.bpo.1-amd64).

Storage driver can be enforced by passing the --storage-driver=overlay option to the Docker daemon.

Configure systemd unit

Docker daemon needs to be configured to start with above options as a systemd service:

$ mkdir /etc/systemd/system/docker.service.d/
$ cat > /etc/systemd/system/docker.service.d/10-execstart.conf << __EOF__
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --storage-driver=overlay --icc=false --iptables=true
__EOF__
$ systemctl daemon-reload && systemctl restart docker

Check if Docker daemon is set up correctly:

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.10.2
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: journald
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 4.3.0-0.bpo.1-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.697 GiB
Name: m1
ID: AAAA:BBBB:AAAA:BBBB:AAAA:BBBB:AAAA:BBBB:AAAA:BBBB:AAAA:BBBB
$ docker run -it --rm debian /bin/echo "WOOHOOO"
Unable to find image 'debian:latest' locally
latest: Pulling from library/debian
fdd5d7827f33: Pull complete 
a3ed95caeb02: Pull complete 
Digest: sha256:e7d38b3517548a1c71e41bffe9c8ae6d6d29546ce46bf62159837aad072c90aa
Status: Downloaded newer image for debian:latest
WOOHOOO

Continue

Multiple Docker hosts with socat tunnels

2016-03-10T00:00:00+01:00

Managing multiple Docker hosts can be done remotely and on-demand with socat tunnels. There is no need for deploying Docker Swarm, reconfiguring Docker daemon, or exposing its port with a proxy.

Socat tunnels to Docker hosts

Docker hosts can be administered either individually or through Docker Swarm. Because Docker daemon does not listen on a network interface by default, a workaround is needed to connect to it remotely.

On all Docker hosts install the socat utility and setup password-less authentication over SSH. On your workstation also install the socat utility and docker command. When needed setup the tunnels with a command like:

$ socat TCP-LISTEN:2350,bind=127.0.0.1,reuseaddr,fork,range=127.0.0.0/8 EXEC:"ssh root@1.2.3.50 socat STDIO UNIX-CONNECT\:/run/docker.sock"
$ for d in 50 51 52; do (socat TCP-LISTEN:23$d,bind=127.0.0.1,reuseaddr,fork,range=127.0.0.0/8 EXEC:"ssh root@1.2.3.$d socat STDIO UNIX-CONNECT\:/run/docker.sock" &); done

Afterwards you may control your Docker hosts from the workstation simply by adding something like -H 127.0.0.1:2350 to the command:

$ iptables -A INPUT -i lo -p tcp -j ACCEPT
$ docker -H 127.0.0.1:2352 ps -a

Weave network driver on Debian 8

2016-07-12T00:00:00+02:00

Weave network driver

Built-in network drivers are great for local communication inside a single host or exposing ports. Unfortunately the built-in multi-host networking support (overlay network driver) is based on VXLAN, which is not encrypted and needs full connectivity between hosts. If you do not have a secure trusted LAN between all hosts, the weave network driver provides a great flexible and secure alternative. Just like Docker daemon, Weave also provides an embedded DNS server for automatic service discovery of containers.

A Weave 1.4.5 network consists of several Docker containers that are managed through a helper script and can be assigned to containers on demand. To build a Weave network we supply addresses of other hosts and the network will automatically (re)connect to peers when they become available.

$ wget -O /usr/local/bin/weave https://git.io/weave
$ chmod +x /usr/local/bin/weave

In case you have a strict firewall DROP policy, you must permit loopback traffic from the weave script (TCP 6784, UDP 53), inter-peer traffic to the Weave control port (TCP 6783), sleeve and fastdp data ports (UDP 6783/6784):

$ iptables -A INPUT -i lo -p tcp --dport=6784 -j ACCEPT
$ iptables -A INPUT -i lo -p udp --dport=53 -j ACCEPT
$ iptables -A INPUT -p tcp --dport=6783 -j ACCEPT
$ iptables -A INPUT -p udp --dport=6783 -j ACCEPT
$ iptables -A INPUT -p udp --dport=6784 -j ACCEPT
$ iptables-save > /etc/iptables/rules.v4

Check if manually starting the Weave network and adding configuration parameters to the docker command, works:

$ weave launch
$ weave status
$ docker $(weave config) run -it --rm debian /bin/ip addr show ethwe
38: ethwe@if39: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether ea:c7:b6:f3:75:da brd ff:ff:ff:ff:ff:ff
    inet 10.32.0.1/12 scope global ethwe
       valid_lft forever preferred_lft forever
    inet6 fe80::e8c7:b6ff:fef3:75da/64 scope link tentative 
       valid_lft forever preferred_lft forever
$ weave reset

Otherwise you may want to create Weave networks on demand (with docker network create --driver=weave mynet) and join containers to them as usual (--net=mynet).

Configure systemd service

Weave network also needs to be configure to start on boot and optionally expose host IP as a systemd service unit:

$ cat > /etc/default/weave << __EOF__
CHECKPOINT_DISABLE=true
CONNLIMIT=100
WEAVE_NO_FASTDP=true
WEAVE_PASSWORD="wfvAwt7sj"
PEERS="1.2.3.4"
__EOF__
$ chmod 600 /etc/default/weave
$ cat > /etc/systemd/system/weave.service << __EOF__
[Unit]
Description=Weave Network
Documentation=http://docs.weave.works/weave/latest_release/
Requires=docker.service
After=docker.service

[Service]
EnvironmentFile=-/etc/default/weave
ExecStartPre=/usr/local/bin/weave launch --no-restart --connlimit \$CONNLIMIT \$PEERS
ExecStart=/usr/bin/docker attach weave
ExecStartPost=/bin/bash -c '/usr/local/bin/weave expose -h \$(hostname -s).weave.local'
ExecStop=/usr/local/bin/weave stop

[Install]
WantedBy=multi-user.target
__EOF__
$ systemctl daemon-reload && systemctl enable weave && systemctl restart weave

Check if Weave network is set up correctly:

$ weave status

        Version: 1.4.5

        Service: router
       Protocol: weave 1..2
           Name: aa:bb:f1:e5:98:a2(foo)
     Encryption: enabled
  PeerDiscovery: enabled
        Targets: 2
    Connections: 3 (2 established, 1 retrying)
          Peers: 3 (with 6 established connections)
 TrustedSubnets: none

        Service: ipam
         Status: ready
          Range: 10.32.0.0-10.47.255.255
  DefaultSubnet: 10.32.0.0/12

        Service: dns
         Domain: weave.local.
       Upstream: 1.2.3.4
            TTL: 1
        Entries: 0

        Service: proxy
        Address: unix:///var/run/weave/weave.sock

        Service: plugin
     DriverName: weave
$ weave status dns
docker-vm    10.45.0.0       weave:expose 82:a1:66:22:11:00

Simulation rs-skip-gram-in-myhdl

2015-10-08T00:00:00+02:00

Simulation rs-skip-gram-in-myhdl implements the skip-gram model with negative sampling (SGNS) in MyHDL.

Computing continuous distributed vector representations of words, also called word embeddings, is becoming increasingly important in natural language processing (NLP). T. Mikolov et al. (2013) introduced the skip-gram model for learning meaningful word embeddings in their word2vec tool. The model takes any text corpus as input, processes pairs of words according to an unsupervised language model, and learns the weights in a custom neural network layer (word embeddings).

There have already been a few attempts at implementing classic neural networks with backpropagation in Verilog or VHDL, but none for word embeddings or in MyHDL that turns Python into a hardware description and verification language.

Open source project:

home: http://gw.tnode.com/student/rs-skip-gram-in-myhdl/
github: http://github.com/gw0/rs-skip-gram-in-myhdl/
technology: Python, MyHDL library

Usage

Requirements:

Python (2.7)
python-virtualenv
auto-installed NumPy (1.8.2)
auto-installed SciPy (0.14.0)
auto-installed MyHDL with fixbv type (on Github branch mep111_fixbv)

Installation on Debian/Ubuntu using virtualenv:

$ apt-get install python python-virtualenv
$ git clone http://github.com/gw0/rs-skip-gram-in-myhdl.git
$ cd ./rs-skip-gram-in-myhdl
$ ./requirements.sh
...
$ . venv/bin/activate

Prepare dataset (example for enwik8-clean.zip):

$ cd ./data
$ wget http://cs.fit.edu/~mmahoney/compression/enwik8.zip
$ unzip enwik8.zip
$ ./clean-wikifil.pl enwik8 > enwik8-clean
$ zip enwik8-clean.zip enwik8-clean
$ rm enwik8.zip enwik8 enwik8-clean
$ cd ..

Execute project (experiment ex01 on dataset data/enwik8-clean.zip):

$ ./project.py ex01 data/enwik8-clean.zip

Implementation

using Python for reading input data
using MyHDL for learning
packing a list of signals to a shadow vector
unpacking a vector to a list of shadow signals
fixed-point numbers (experimental fixbv type, on Github branch mep111_fixbv)
- minimal number: -2^7
- maximal number: 2^7
- resolution: 2^-8
- total bits: 16
skip-gram model
- with negative sampling with ratio 1:1
- word embedding vector size: 3
- ReLU activation function with leaky factor: 0.01
- constant learning rate: 0.1
- initial word embedding spread: 0.1
- exponential moving average of mean square error with factor: 0.01

Components:

project.py - Main code for preparing real input data and passing it to training stimulus.
train.py - Training stimulus of skip-gram model with negative sampling (SGNS).
RamSim.py - Simulated RAM model using a Python dictionary.
Rectifier.py - Rectified linear unit (ReLU) activation function model using fixbv type.
DotProduct.py - Vector dot product model using fixbv type.
WordContextProduct.py - Word-context embeddings product model needed for skip-gram training.
WordContextUpdated.py - Word-context embeddings updated model needed for skip-gram training.

Testing components

$ python RamSim.py
 10 write, addr: 0, din: 0
 20 write, addr: 1, din: 2
 30 write, addr: 2, din: 4
 40 write, addr: 3, din: 6
 50 write, addr: 4, din: 8
 60 read, addr: 0, dout: 0
 70 read, addr: 1, dout: 2
 80 read, addr: 2, dout: 4
 90 read, addr: 3, dout: 6
100 read, addr: 4, dout: 8

$ python Rectifier.py
 20 x: -2.500000, y: -0.031250, y_dx: 0.011719
 30 x: -2.000000, y: -0.023438, y_dx: 0.011719
 40 x: -1.500000, y: -0.019531, y_dx: 0.011719
 50 x: -1.000000, y: -0.011719, y_dx: 0.011719
 60 x: -0.500000, y: -0.007812, y_dx: 0.011719
 70 x: 0.000000, y: 0.000000, y_dx: 0.011719
 80 x: 0.500000, y: 0.500000, y_dx: 1.000000
 90 x: 1.000000, y: 1.000000, y_dx: 1.000000
100 x: 1.500000, y: 1.500000, y_dx: 1.000000
110 x: 2.000000, y: 2.000000, y_dx: 1.000000

$ python DotProduct.py
 20 a_list: [-2.0, 0.0, 0.0], b_list: [0.0, 0.0, 0.0], y: 0.000000, y_da: [0.0, 0.0, 0.0], y_db: [-2.0, 0.0, 0.0]
 30 a_list: [-1.5, 0.0, 0.0], b_list: [0.5, 0.0, 0.0], y: -0.750000, y_da: [0.5, 0.0, 0.0], y_db: [-1.5, 0.0, 0.0]
 40 a_list: [-1.0, 0.0, 0.0], b_list: [1.0, 0.0, 0.0], y: -1.000000, y_da: [1.0, 0.0, 0.0], y_db: [-1.0, 0.0, 0.0]
 50 a_list: [-0.5, 0.0, 0.0], b_list: [1.5, 0.0, 0.0], y: -0.750000, y_da: [1.5, 0.0, 0.0], y_db: [-0.5, 0.0, 0.0]
 60 a_list: [0.0, 0.0, 0.0], b_list: [2.0, 0.0, 0.0], y: 0.000000, y_da: [2.0, 0.0, 0.0], y_db: [0.0, 0.0, 0.0]
 70 a_list: [0.5, 0.0, 0.0], b_list: [2.5, 0.0, 0.0], y: 1.250000, y_da: [2.5, 0.0, 0.0], y_db: [0.5, 0.0, 0.0]
 80 a_list: [1.0, 0.0, 0.0], b_list: [3.0, 0.0, 0.0], y: 3.000000, y_da: [3.0, 0.0, 0.0], y_db: [1.0, 0.0, 0.0]
 90 a_list: [1.5, 0.0, 0.0], b_list: [3.5, 0.0, 0.0], y: 5.250000, y_da: [3.5, 0.0, 0.0], y_db: [1.5, 0.0, 0.0]
100 a_list: [2.0, 0.0, 0.0], b_list: [4.0, 0.0, 0.0], y: 8.000000, y_da: [4.0, 0.0, 0.0], y_db: [2.0, 0.0, 0.0]
110 a_list: [2.5, 0.0, 0.0], b_list: [4.5, 0.0, 0.0], y: 11.250000, y_da: [4.5, 0.0, 0.0], y_db: [2.5, 0.0, 0.0]

$ python WordContextProduct.py
 20 word: [-2.0, 0.0, 0.0], context: [0.0, 0.0, 0.0], y: 0.000000, y_dword: [0.0, 0.0, 0.0], y_dcontext: [-0.0234375, 0.0, 0.0]
 30 word: [-1.5, 0.0, 0.0], context: [0.5, 0.0, 0.0], y: -0.007812, y_dword: [0.0078125, 0.0, 0.0], y_dcontext: [-0.01953125, 0.0, 0.0]
 40 word: [-1.0, 0.0, 0.0], context: [1.0, 0.0, 0.0], y: -0.011719, y_dword: [0.01171875, 0.0, 0.0], y_dcontext: [-0.01171875, 0.0, 0.0]
 50 word: [-0.5, 0.0, 0.0], context: [1.5, 0.0, 0.0], y: -0.007812, y_dword: [0.01953125, 0.0, 0.0], y_dcontext: [-0.0078125, 0.0, 0.0]
 60 word: [0.0, 0.0, 0.0], context: [2.0, 0.0, 0.0], y: 0.000000, y_dword: [0.0234375, 0.0, 0.0], y_dcontext: [0.0, 0.0, 0.0]
 70 word: [0.5, 0.0, 0.0], context: [2.5, 0.0, 0.0], y: 1.250000, y_dword: [2.5, 0.0, 0.0], y_dcontext: [0.5, 0.0, 0.0]
 80 word: [1.0, 0.0, 0.0], context: [3.0, 0.0, 0.0], y: 3.000000, y_dword: [3.0, 0.0, 0.0], y_dcontext: [1.0, 0.0, 0.0]
 90 word: [1.5, 0.0, 0.0], context: [3.5, 0.0, 0.0], y: 5.250000, y_dword: [3.5, 0.0, 0.0], y_dcontext: [1.5, 0.0, 0.0]
100 word: [2.0, 0.0, 0.0], context: [4.0, 0.0, 0.0], y: 8.000000, y_dword: [4.0, 0.0, 0.0], y_dcontext: [2.0, 0.0, 0.0]
110 word: [2.5, 0.0, 0.0], context: [4.5, 0.0, 0.0], y: 11.250000, y_dword: [4.5, 0.0, 0.0], y_dcontext: [2.5, 0.0, 0.0]

$ python WordContextUpdated.py
 20 word: [-2.0, 0.0, 0.0], context: [0.0, 0.0, 0.0], mse: 1.000000, y: 0.000000, new_word: [-2.0, 0.0, 0.0], new_context: [-0.00390625, 0.0, 0.0]
 30 word: [-1.5, 0.0, 0.0], context: [0.5, 0.0, 0.0], mse: 1.015625, y: -0.007812, new_word: [-1.5, 0.0, 0.0], new_context: [0.49609375, 0.0, 0.0]
 40 word: [-1.0, 0.0, 0.0], context: [1.0, 0.0, 0.0], mse: 1.023438, y: -0.011719, new_word: [-1.0, 0.0, 0.0], new_context: [1.0, 0.0, 0.0]
 50 word: [-0.5, 0.0, 0.0], context: [1.5, 0.0, 0.0], mse: 1.015625, y: -0.007812, new_word: [-0.49609375, 0.0, 0.0], new_context: [1.5, 0.0, 0.0]
 60 word: [0.0, 0.0, 0.0], context: [2.0, 0.0, 0.0], mse: 1.000000, y: 0.000000, new_word: [0.00390625, 0.0, 0.0], new_context: [2.0, 0.0, 0.0]
 70 word: [0.5, 0.0, 0.0], context: [2.5, 0.0, 0.0], mse: 0.062500, y: 1.250000, new_word: [0.4375, 0.0, 0.0], new_context: [2.48828125, 0.0, 0.0]
 80 word: [1.0, 0.0, 0.0], context: [3.0, 0.0, 0.0], mse: 4.000000, y: 3.000000, new_word: [0.390625, 0.0, 0.0], new_context: [2.796875, 0.0, 0.0]
 90 word: [1.5, 0.0, 0.0], context: [3.5, 0.0, 0.0], mse: 18.062500, y: 5.250000, new_word: [-0.01171875, 0.0, 0.0], new_context: [2.8515625, 0.0, 0.0]
100 word: [2.0, 0.0, 0.0], context: [4.0, 0.0, 0.0], mse: 49.000000, y: 8.000000, new_word: [-0.84375, 0.0, 0.0], new_context: [2.578125, 0.0, 0.0]
110 word: [2.5, 0.0, 0.0], context: [4.5, 0.0, 0.0], mse: 105.062500, y: 11.250000, new_word: [-2.18359375, 0.0, 0.0], new_context: [1.8984375, 0.0, 0.0]

 10 mse: 0.992188, y: 0.003906, word: [0.0625, 0.02734375, 0.07421875], context: [0.00390625, 0.0234375, 0.06640625]
 20 mse: 0.984375, y: 0.007812, word: [0.0625, 0.03125, 0.08203125], context: [0.01171875, 0.02734375, 0.07421875]
 30 mse: 0.984375, y: 0.007812, word: [0.0625, 0.03515625, 0.08984375], context: [0.01953125, 0.03125, 0.08203125]
...
100 mse: 0.921875, y: 0.039062, word: [0.09375, 0.0625, 0.16796875], context: [0.07421875, 0.05859375, 0.16796875]
 ...
200 mse: 0.597656, y: 0.226562, word: [0.20703125, 0.1484375, 0.40234375], context: [0.19921875, 0.1484375, 0.40234375]
 ...
300 mse: 0.097656, y: 0.687500, word: [0.35546875, 0.26171875, 0.703125], context: [0.3515625, 0.26171875, 0.703125]
...
400 mse: 0.003906, y: 0.949219, word: [0.421875, 0.30859375, 0.82421875], context: [0.41796875, 0.30859375, 0.82421875]
410 mse: 0.000000, y: 0.960938, word: [0.42578125, 0.30859375, 0.828125], context: [0.421875, 0.30859375, 0.828125]

$ python train.py
   40 1 mse_ema: 1.000000, mse: 0.984375, word: [0.09375, 0.0625, 0.0], context: [0.06640625, 0.0546875, 0.07421875]
  100 1 mse_ema: 1.000000, mse: 0.000000, word: [0.1015625, 0.06640625, 0.0078125], context: [0.05078125, 0.0234375, 0.05859375]
  160 1 mse_ema: 0.988281, mse: 0.992188, word: [0.05859375, 0.03125, 0.01953125], context: [0.0625, 0.04296875, 0.0234375]
  220 1 mse_ema: 0.988281, mse: 0.000000, word: [0.06640625, 0.03515625, 0.0234375], context: [0.0234375, 0.03515625, 0.06640625]
  280 1 mse_ema: 0.976562, mse: 0.984375, word: [0.0859375, 0.0390625, 0.0], context: [0.0546875, 0.0625, 0.0078125]
  340 1 mse_ema: 0.976562, mse: 0.000000, word: [0.08984375, 0.046875, 0.0], context: [0.09765625, 0.01953125, 0.03515625]
...
 1960 1 mse_ema: 0.816406, mse: 0.992188, word: [0.0, 0.0546875, 0.015625], context: [0.03515625, 0.0859375, 0.05859375]
 2020 1 mse_ema: 0.820312, mse: 0.000000, word: [0.00390625, 0.0625, 0.0234375], context: [0.00390625, 0.00390625, 0.0234375]
...
10000 5 mse_ema: 0.554688, mse: 0.945312, word: [0.12890625, 0.09765625, 0.06640625], context: [0.12109375, 0.046875, 0.09765625]
10060 5 mse_ema: 0.558594, mse: 0.000000, word: [0.140625, 0.1015625, 0.07421875], context: [0.04296875, 0.0703125, 0.06640625]
...
20080 9 mse_ema: 0.492188, mse: 0.871094, word: [0.01953125, 0.23828125, 0.1484375], context: [0.01171875, 0.1953125, 0.12109375]
20140 9 mse_ema: 0.496094, mse: 0.000000, word: [0.01953125, 0.2578125, 0.16015625], context: [0.02734375, 0.0234375, 0.0078125]
...
30040 14 mse_ema: 0.457031, mse: 0.835938, word: [0.17578125, 0.140625, 0.18359375], context: [0.1796875, 0.140625, 0.19140625]
30100 14 mse_ema: 0.460938, mse: 0.000000, word: [0.19140625, 0.15234375, 0.203125], context: [0.01953125, 0.0390625, 0.0546875]
...
40000 18 mse_ema: 0.328125, mse: 0.628906, word: [0.16796875, 0.35546875, 0.234375], context: [0.17578125, 0.3515625, 0.21875]
40060 18 mse_ema: 0.332031, mse: 0.000000, word: [0.18359375, 0.3828125, 0.25390625], context: [0.05078125, 0.03125, 0.078125]
...
50020 22 mse_ema: 0.164062, mse: 0.015625, word: [0.23828125, 0.6171875, 0.8046875], context: [0.046875, 0.05859375, 0.09765625]
50080 22 mse_ema: 0.164062, mse: 0.082031, word: [0.01953125, 0.8671875, 0.53515625], context: [0.01953125, 0.59765625, 0.3671875]
50140 22 mse_ema: 0.164062, mse: 0.003906, word: [0.01953125, 0.8828125, 0.546875], context: [0.09375, 0.02734375, 0.06640625]
50200 23 mse_ema: 0.164062, mse: 0.285156, word: [0.5078125, 0.38671875, 0.234375], context: [0.5078125, 0.38671875, 0.23828125]

Packing a list of signals to a shadow vector

MyHDL does not support conversion of a list of signals as a port to a module. Attempting convert them to Verilog or VHDL results in:

myhdl.ConversionError: in file DotProduct.py, line 14:
    List of signals as a port is not supported: y_da_list

Instead of manipulating bits directly shadow signals provide a read-only higher level abstraction. It is also possible to use them to cast between types.

Lets suppose in the test bench you manipulate a list of signals a_list (read-write) and want to pass them all to your module (read-only). To make such code convertible the list of signals must be concatenated into a shadow vector signal.

    a_list = [ Signal(fixbv(0.0, min=fix_min, max=fix_max, res=fix_res)) for _ in range(dim) ]
    a_vec = ConcatSignal(*reversed(a_list))
    foo = Foo(y, a_vec)

Inside your module Foo(y, a_vec) you may want to access individual signals again, so you must assign slices of the vector back to a list of shadow signals.

    a_list = [ Signal(fixbv(0.0, min=fix_min, max=fix_max, res=fix_res)) for j in range(dim) ]
    for j in range(dim):
        a_list[j].assign(a_vec((j + 1) * fix_width, j * fix_width))

Unpacking a vector to a list of shadow signals

Lets suppose your module outputs a vector y_vec (read-write), but in your test bench you want to process them as individual signals (read-only). For a convertible code first prepare a sufficiently wide bitwise signal and then assign slices of it into a list of shadow signals y_list.

    y_vec = Signal(intbv(0)[dim * fix_width:])
    y_list = [ Signal(fixbv(0.0, min=fix_min, max=fix_max, res=fix_res)) for j in range(dim) ]
    for j in range(dim):
        y_list[j].assign(y_vec((j + 1) * fix_width, j * fix_width))
    foo = Foo(y_vec, a)

Assigning values inside your module Foo(y_vec, a) is more troublesome, but can be accomplished by doing bitwise assignments at correct offsets.

    tmp = fixbv(123.0, min=fix_min, max=fix_max, res=fix_res)
    y_vec.next[(j + 1) * fix_width:j * fix_width] = tmp[:]

Feedback

Unfortunately development past current execution in MyHDL simulator is not planned. But in case you fix any bugs or develop new features, feel free to submit a pull request on GitHub.

License

This code is licensed under the GNU Affero General Public License 3.0+ (AGPL-3.0+). Note that it is mandatory to make all modifications and complete source code publicly available to any user.

Learning Representations for Text-level Discourse Parsing

2015-07-28T00:00:00+02:00

Docker ubuntu-systemd

2016-06-16T00:00:00+02:00

ubuntu-systemd is a minimal Docker image built from Ubuntu 15.04 with systemd designed for running in an unprivileged container. Main philosophy:

simple to use and maintain, same system management experience
transparent build process, unlike “official” Ubuntu images
treat containers as VMs, multiple processes inside a single container

You can use it as a base for your own Docker images. Just pull it from the Docker hub.

Open source project:

home: http://gw.tnode.com/docker/ubuntu-systemd/
github: http://github.com/tozd/docker-ubuntu-systemd/
technology: ubuntu, systemd
docker hub: https://hub.docker.com/r/tozd/ubuntu-systemd/

Usage

To run systemd in an unprivileged container a few manual tweaks are currently necessary. Remember to replace /tmp in following commands to a non-world-readable directory.

First it depends on the cgroups directory, at least it needs read-only access to cgroup name=systemd hierarchy (in /sys/fs/cgroup/systemd). Lets prepare one for all ubuntu-systemd containers:

$ mkdir -p /tmp/cgroup/systemd && mount -t cgroup systemd /tmp/cgroup/systemd -o ro,noexec,nosuid,nodev,none,name=systemd

# or alternatively:
$ mkdir -p /tmp/cgroup/systemd && mount --bind /sys/fs/cgroup/systemd /tmp/cgroup/systemd

Next it needs tmpfs mount points in /run and /run/lock. This needs to be prepared separately for each ubuntu-systemd container:

$ mkdir /tmp/run && mount -t tmpfs tmpfs /tmp/run
$ mkdir /tmp/run/lock && mount -t tmpfs tmpfs /tmp/run/lock

You could also add all mount points permanently to your /etc/fstab:

systemd  /tmp/cgroup/systemd  cgroup  ro,noexec,nosuid,nodev,none,name=systemd  0  0
tmpfs  /tmp/run  tmpfs  nodev,nosuid,mode=755,size=65536k  0  0
tmpfs  /tmp/run/lock  tmpfs  nodev,nosuid,mode=755,size=65536k  0  0

Then you are ready to use your Docker container:

$ docker run -d --name xxx -v /tmp/cgroup:/sys/fs/cgroup:ro -v /tmp/run:/run:rw tozd/ubuntu-systemd
$ docker exec -it xxx /bin/bash

Please note that graceful stopping and removal of the Docker container looks a little different now:

$ docker kill --signal SIGPWR xxx && docker stop xxx

$ docker rm -f xxx
$ umount /tmp/run/lock /tmp/run && rmdir /tmp/run
$ umount /tmp/cgroup/systemd && rmdir /tmp/cgroup/systemd /tmp/cgroup

Build

Build `ubuntu-systemd`

All instructions from scratch are included in the Dockerfile. To build it you just need to run:

$ git clone http://github.com/tozd/docker-ubuntu-systemd.git
$ docker build -t tozd/ubuntu-systemd -t tozd/ubuntu-systemd:15.04.0 ./docker-ubuntu-systemd

Build `debootstrap-minbase.tgz`

The standard debootstrap tool is used to generate the initial minimal Ubuntu system. As we are using Docker, we can build the base image there without installing anything on the host:

$ mkdir /tmp/ubuntu-systemd
$ docker run -it --rm --privileged -v /tmp/ubuntu-systemd:/mnt ubuntu /bin/bash

$ cd /mnt
$ apt-get update && apt-get install -y debootstrap

$ debootstrap --variant=minbase --components=main vivid ./rootfs
$ rm -f ./rootfs/var/cache/apt/archives/*.deb ./rootfs/var/cache/apt/archives/partial/*.deb ./rootfs/var/cache/apt/*.bin

$ tar --numeric-owner -zcf "debootstrap-minbase.tgz" -C "./rootfs" . && rm -rf "./rootfs"
$ exit

Feedback

If you encounter any bugs or have feature requests, please file them in the issue tracker or even develop it yourself and submit a pull request over GitHub.

License

Learning Representations for Text-level Discourse Parsing

2015-08-21T00:00:00+02:00

Conference proceeding

G. Weiss, “Learning Representations for Text-level Discourse Parsing,” in Proceedings of the ACL-IJCNLP 2015 Student Research Workshop, 2015, pp. 16–21.

Abstract

In the proposed doctoral work we will design an end-to-end approach for the challenging NLP task of text-level discourse parsing. Instead of depending on mostly hand-engineered sparse features and independent components for each subtask, we propose a unified approach completely based on deep learning architectures. To train better dense vector representations that capture communicative functions and semantic roles of discourse units and relations between them, we will jointly learn all discourse parsing subtasks at different layers of our stacked architecture and share their intermediate representations. By combining unsupervised training of word embeddings and related NLP tasks with our guided layer-wise multi-task learning of higher representations we hope to reach or even surpass performance of current state-of-the-art methods on annotated English corpora.

Issues and workarounds for Debian 8

2015-05-05T00:00:00+02:00

Debian 8 brings updates to 66% of packages and switches to systemd as its default system and service manager. This can result in a couple of issues, so check below for workarounds and solutions.

Systemd ignores `decrypt_derived` keyscript

Many security aware users of Debian have encrypted root, swap, home, and other partitions using cryptsetup. Automatic mounting of those partitions at boot time can be configured in /etc/crypttab. Instead of entering passwords for each partition each time it is possible to use keyscript scripts to use other key sources and automate the process. One of those keyscripts is decrypt_derived that provides a way to chain encrypted partitions, such that the encryption key is automatically derived from a previously unlocked partition.

This used to work pretty well on Debian 7 and older that used SysV service manager:

$ cat /etc/crypttab
debian_crypt UUID=08bd04d5-... none luks
sdb1_crypt UUID=a84f890c-... debian_crypt luks,keyscript=decrypt_derived
swap_crypt /dev/sda2 debian_crypt swap,cipher=aes-cbc-essiv:sha256,hash=ripemd160,size=256,keyscript=decrypt_derived

Although the new systemd service manager processes the /etc/crypttab configuration, it unfortunately ignores keyscripts and can not use anything equivalent.

$ cat /var/log/syslog
Apr 04 20:37:38 xxx systemd-cryptsetup[544]: Encountered unknown
/etc/crypttab option 'keyscript=/lib/cryptsetup/scripts/decrypt_derived', ignoring.

Workaround

Instead of using decrypt_derived keyscript, you can save its output into a file and put it on the encrypted partition from which it derived. For the above example, where debian_crypt is the root partition and other keys are derived from it, this means saving it into /.debian_crypt.key (make sure no one but root can access it). Then specify this as the key file, add a dependency to initially mount this partition, and do not forget to update the boot-time initramfs archive. This at least works for the encrypted root partition on which others depend on and is as secure as using the decrypt_derived keyscript.

$ /lib/cryptsetup/scripts/decrypt_derived debian_crypt > /.debian_crypt.key
$ chown root:root /.debian_crypt.key
$ chmod 600 /.debian_crypt.key
$ cat /etc/crypttab
debian_crypt UUID=08bd04d5-... none luks
sdb1_crypt UUID=a84f890c-... /.debian_crypt.key luks
swap_crypt /dev/sda2 /.debian_crypt.key swap,cipher=aes-cbc-essiv:sha256,hash=ripemd160,size=256
$ grep CRYPTDISKS_MOUNT /etc/default/cryptdisks
CRYPTDISKS_MOUNT="/dev/mapper/debian_crypt"
$ update-initramfs -u
$ reboot

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=618862

Systemd hibernation with encrypted swap partition

Although suspend to RAM works well on all versions of Debian, one may also want to hibernate it to disk now and then. Unfortunately this does not work if you are using an encrypted swap partition. It seems that by digging into the initramfs archive scripts and hibernation hooks one could accomplish this in Debian 7 or older, but in Debian 8 with systemd manager it seems to become more complicated.

Nvidia graphic driver kernel updates

Proprietary kernel drivers and apps often cause problems and inconveniences. One of them is recompiling the binary driver each time you switch to a new kernel. If you fail to do that, you will not be able to start the X.org server graphical window interface and be stuck in the console.

Workaround

Luckily Debian 8 or older contain a Nvidia DKMS driver that, when installed, automatically recompiles itself after each kernel update.

$ aptitude install nvidia-kernel-dkms xserver-xorg-video-nvidia

Skype error loading `libGL.so.1`

In case you are using Skype on a 64-bit system, it may not even start after updating to Debian 8. Skype is looking for an i386 library that is not provided by the Nvidia package.

$ skype
libGL.so.1: cannot open shared object file: No such file or directory

Workaround

As an workaround the MESA i386 library libGL1.so.1 can be used. To use this library for Skype create the file /etc/ld.so.conf.d/skype.conf with the following contents and reload shared libraries.

$ cat /etc/ld.so.conf.d/skype.conf
# Skype error while loading shared libraries: libGL.so.1: cannot open shared object file: No such file or directory
/usr/lib/mesa-diverted/i386-linux-gnu
$ ldconfig -v

http://askubuntu.com/questions/257897/error-loading-libgl-so-1

Black screen with cursor after suspend

When using a proprietary Nvidia driver, it may happen it doesn’t initialize correctly after returning from a suspend to RAM, so you end up at a black or garbled screen with a cursor. The problem seems to be connected with using OpenGL effects on a composing desktop environment.

Workaround

In KDE under System Settings/Workspace Appearance and Behavior/Desktop Effects those desktop effects can be turned off. Alternatively you can setup a keyboard shortcut (default Alt+Shift+F12) to toggle effects and each time you get stuck in a black screen just switch it off and on again, so that everything gets redrawn correctly.

KDE screensaver during a fullscreen application

In KDE 4.11 it sometimes happens that the screensaver turns on although you are watching a movie in fullscreen, e.g. in VLC media player, from browser with Flash or HTML5 video. In previous versions a workaround script called lightsOn.sh used to prevent the screensaver from turning on by triggering events while it was running.

Workaround

So the only way to is to actually turn it off. In KDE under System Settings/Hardware/Display and Monitor/Screen Locker turn off Start automatically after # minutes and turn it back on when you stop watching the fullscreen application.

You may also try programmatically disabling and enabling the screensaver with:

$ xset -dpms; xset s off
$ xset +dpms; xset s on

http://askubuntu.com/questions/193930/how-to-disable-sleep-screensaver-kubuntu-12-04-lts-and-vlc

Unbootable with software RAID1 and GRUB2

There is a bug in Debian Installer that results in an unbootable system after fresh installation of Debian 8.1. It happens if you manually partition your disks and setup all partitions in software RAID array, including partition with /boot. mdadm will by default create arrays with metadata format version 1.2 (not 0.90). This messes up the GRUB2 installation step so that it doesn’t install itself into the master boot record of hard drives correctly. As a result the system hangs and shows a blank screen where GRUB2 boot loader should appear.

Workaround

During the installation process, just after you Install the GRUB boot loader on a hard disk, you should switch to the console (Ctrl+Alt+F2) and force install GRUB2 boot loader on all hard drives (not partitions) and update initramfs (in case it was missing something):

$ mount -t proc proc /target/proc
$ chroot /target
$ update-grub
$ grub-install --recheck /dev/sda
$ grub-install --recheck /dev/sdb
$ update-initramfs -u

Afterwards switch back to the Debian Installer (Ctrl+Alt+F1) and finalize the installation.

http://www.howtoforge.com/how-to-set-up-software-raid1-on-a-running-system-incl-grub2-configuration-debian-squeeze-p2

DKMS could not locate `dkms.conf`

Each time you upgrade the kernel, you should check whether everything went smooth, especially if you are using proprietary drivers or custom kernel modules. It can happen that the DKMS framework for automatically recompiling kernel modules gets stuck and consequently some modules won’t work under the new kernel.

$ dkms status
Error! Could not locate dkms.conf file.
File:  does not exist.

Cause of this error is an invalid state in DKMS build directories /var/lib/dkms.

Workaround

A simple solution is to manually remove all DKMS build directories and reinstall all *-dkms packages.

$ rm -rf /var/lib/dkms/nvidia /var/lib/dkms/nvidia-current /var/lib/dkms/virtualbox

$ aptitude search '~i-dkms'
i   nvidia-kernel-dkms
i   virtualbox-dkms
$ aptitude reinstall nvidia-kernel-dkms
...
Setting up nvidia-kernel-dkms (340.65-2) ...
Loading new nvidia-current-340.65 DKMS files...
Building only for 3.16.0-4-amd64
Building initial module for 3.16.0-4-amd64
Done.
nvidia-current:
...
nvidia-uvm.ko:
...
DKMS: install completed.
$ aptitude reinstall virtualbox-dkms
...
Setting up virtualbox-dkms (4.3.18-dfsg-3+deb8u3) ...
Loading new virtualbox-4.3.18 DKMS files...
Building only for 3.16.0-4-amd64
Building initial module for 3.16.0-4-amd64
Done.
vboxdrv:
...
vboxnetadp.ko:
...
vboxnetflt.ko:
...
vboxpci.ko:
...
DKMS: install completed.

Afterwards make sure your kernel modules were recompiled for the new kernel:

$ dkms status
nvidia-current, 340.65, 3.16.0-4-amd64, x86_64: installed
virtualbox, 4.3.18, 3.16.0-4-amd64, x86_64: installed

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695824

[Sem3] Generic feature extraction for text categorization

2015-01-08T00:00:00+01:00

[Sem3] Generic feature extraction for text categorization

2015-01-04T00:00:00+01:00

Student paper

Related work review of “Generic feature extraction for text categorization”.

Abstract

Extracting more informative text features improves the performance of text mining tasks, such as text categorization and sentiment analysis. Nevertheless researchers usually focus on the main part of their problem and just preprocess texts with well-known linguistic and custom methods, that are based on background knowledge about a given language, domain, and specific task. To improve and automate the initial preprocessing phase we propose a generic feature extraction method inspired by inductive programming. Beginning almost without any natural language processing knowledge it will heuristically try to define and combine elemental feature extractors until more complex and informative features are found.

Screenshots for glyphs

2015-08-24T00:00:00+02:00

Screenshots for glyphs (formerly called Screenshots for Ingress) is an integrated glyph hacking recorder app for games, such as Ingress–the popular augmented reality massively multiplayer online role-playing mobile game for Android.

News

new version 0.9 (2015-08-24)
fix integration with Ingress on Android 5.1.1 and M
old: temporarily removed from Google Play, as Niantic Labs filed a complaint without explaining what exactly is violated

Usage

Screenshots for glyphs works as a simple floating application that displays the last few screenshots taken. It can therefore be useful for glyph hacking in games, such as Ingress (in Portal view long-touch the Hack button).

On stock phones press the usual screenshot trigger combination (simultaneously press for a second or two one of: power + volume down, or power + home). Unfortunately this method can be too slow, use screenshots only for one or two glyphs and your brain for the rest.
On rooted phones screenshots can be taken programmatically much faster just by touching the floating icon. Make sure you have working binaries /system/xbin/su and /system/bin/screencap.
Long-touch the floating icon to remove the screenshots and shrink the window.

Notice

Make sure you have the game Ingress installed. This recorder app automatically launches Ingress and shows a floating icon somewhere in the top-right corner. If you close Ingress or switch to another application, the icon disappears as if it were integrated into the game.

Warning! Screenshots are automatically deleted after being taken as long as the application is running.

Play Ingress, join Enlightened! Please spread this app to all Ingress players and give 5 stars if you like it. Thank you!

Resistance players: The app contains the green logo and a notice. Please donate if you want to see this changed. Otherwise giving negative reviews will encourage us to block it to one faction only.

Other

Donations & Crowdfunding

http://gw.tnode.com/donations/

To support the development of free apps and release them under an open-source license we encourage you to donate a small amount. If every user would pay just $1, this would result in around $1500 of donations. When the amount is reached, all source code will be publicly released. Accepted payment methods are PayPal, credit cards, and bitcoins.

Feedback

If you encounter any bugs or have feature requests, please send an and we’ll see what we can do. Please give 5 stars or donate if you like it. Thank you!

License

Screenshots for glyphs is NOT affiliated in any way with Google Inc., Niantic Labs or Ingress. It is NOT intended for sole use with Ingress and no graphics or other material from Ingress was abused for promotional purposes.

Screenshots for glyphs does not hack or interfere in any way with the normal behavior of any glyph-hacking game and is not intended to do so. It only helps at the process of remembering glyphs, you still need to create the screenshots yourself, and draw the glyphs.

Screenshots

Screenshots in action

Screenshots before action

Screenshots in game

Screenshots in action again

CyanogenMod 11 on Nexus 5

2014-08-10T00:00:00+02:00

LG Nexus 5 is a powerful mobile phone and stock version of Android has nearly everything. Nevertheless CyanogenMod 11 M9 (or newer), based on Android 4.4.4 (KitKat), can improve the experience, security, and customization even further.

Preparation

Requirements

LG Nexus 5 (Google/LG D820/D821, hammerhead)
micro USB cable
backup your data (contacts, calendar, photos, videos…)
turn off phone encryption (if you enabled it, as it causes problems)
working adb and fastboot from Android SDK Tools

Download

Android SDK Tools and Platform-tools
ClockworkMod Recovery 6.0.4.5 (or TWRP Recovery 2.8.7.0 or newer) for LG Nexus 5
CyanogenMod 11 M9 (or newer) snapshot for LG Nexus 5
Google Apps for CM 11

Installation

Installing CyanogenMod 11 on LG Nexus 5 is straightforward as there is actually no need to exploit a security hole in the system. There is also no need to create backup images of your original stock Android, as all factory versions are officially available.

For a thorough description see the usual CyanogenMod 11 installation instructions, but the following steps should be sufficient.

Unlocking

Reboot into the bootloader using USB (alternatively volume up + volume down + power) and unlock the bootloader:

$ adb reboot bootloader
$ fastboot oem unlock

In case of permission problems, run commands as root and enable USB debugging on the device. On success an unlocked icon will appear at the bottom of the Google boot screen during reboots.

Recovery console

Again reboot into the bootloader (as before) and flash previously downloaded advanced recovery console ClockworkMod Recovery (or TWRP Recovery):

$ adb reboot bootloader
$ fastboot flash recovery recovery-clockwork-6.0.4.5-hammerhead.img

Wait for the flashing procedure to complete.

Install CyanogenMod

Reboot into the recovery using USB (alternatively volume up + volume down + power, navigate with volume up/down, and confirm using power button):

$ adb reboot recovery

In ClockworkMod Recovery select wipe data/factory reset (navigate with volume up/down, and confirm using power button). Repeat procedure for wipe cache partition. In case of problems with encryption also navigate to mounts and storage and format /data.

Afterwards transfer and install previously downloaded CyanogenMod 11 package by selecting install zip and using the install zip from sideload method. Transfer the package over USB with:

$ adb sideload cm-11-20140805-SNAPSHOT-M9-hammerhead.zip

Wait for the flashing and installation to complete. The procedure is complete if there were no fatal error messages and you regained control over the menu on top.

Install Google Apps

As CyanogenMod 11 comes without any apps from Google (especially without Google Play Store) it is recommended to install them too.

Again reboot into the recovery (if not there yet) and install previously downloaded Google Apps for CM 11 by selecting install zip and using the install zip from sideload method:

$ adb reboot recovery
$ adb sideload gapps-kk-20140606-signed.zip

Setup wizard

Once the installation has finished, you can select reboot system now and wait for it to boot the first time (takes a few minutes).

A setup wizard will appear and it is recommended to not setup a Google account yet, as it can not be configured in detail here and it initiates syncing and restoration procedures.

Security

Encrypt phone

Newest Android phones also support a full-device encryption feature to prevent potential thieves from accessing your data. Note that encryption decreases the performance a little and is one-way only (factory reset to turn it off).

Unfortunately by default the boot-time encryption password is linked with your lock screen PIN or password (other options are unavailable), which may be inconvenient if you are used to an unlock pattern. Nevertheless it is possible to manually enable encryption independent of your lock screen.

For this first open Settings/Lock screen/Screen security and setup your preferred Screen lock method (such as Pattern). Choose a good pattern as it is not trivial to change it later on.

Than enable encryption independently on a rooted system through the terminal (choose a long secure password). In case you are using USB you will need to temporarily enable Settings/Superuser/Root access to Apps and ADB to gain root privileges (turn it off afterwards):

$ adb shell
$ su
$ vdc cryptfs enablecrypto inplace [password]

After a few seconds your Android device will reboot and begin with the encryption process (takes around 30 min). On completion your system will reboot and now you will only need to enter the chosen password once for each reboot.

Firewall

Install AFWall+ (Android Firewall +) and allow superuser access forever. It is recommended to enable the following Preferences:

check Enable notifications
check Active Rules
check LAN Control
check IPv6 support (in case of problems copy systems ip6tables to app directory)
check Confirm box on AFWall+ disable (to prevent accidentally disabling)
check Enable Firewall Logs

Do not forget to Enable firewall and Apply. Now use apps normally and check logs if any app has internet connection issues (than enable access).

Note that CM Updater does not work with CWM 6.0.4.5 therefore there is no need to allow internet access for Settings and CM Updater.

In case Miracast display functionality is not working, you may need to use the following custom startup script:

$IPTABLES -A "droidwall-wifi" -p udp --destination 224.0.0.0/16 -j RETURN
$IPTABLES -A "droidwall-lan" -p udp -j RETURN
$IPTABLES -A "droidwall-input" -p udp --source 192.168.0.0/16 -j RETURN

Install and update

You will probably want to setup Google Play Store to enable simple installation and updating of apps.

Apps you really want to install:

AFWall+ (Android Firewall +)
LostNet NoRoot Firewall (interesting alternative)
Google Camera
Titanium Backup

Also check settings of apps:

Google Play Store - in Settings: disable auto-update apps
YouTube - in Settings/General: turn off Improve YouTube, check Limit mobile data usage
Hangouts - turn off Improve
Google+
Google Maps - turn off Shake to send feedback
Phone - in Settings/Advanced: turn off Call lookup (turn off Caller ID by Google, turn off Nearby places)
Google Translate - in Settings/Data usage: turn off Prefer network text-to-speach, turn off Improve camera input

Privacy settings

Many security and privacy options are available and you may want to check them out under Settings:

Data usage: set mobile data limit, enable show Wi-Fi usage
Mobile networks/Access Point Names/Internet APN: remove proxy setting
Wireless & networks/More…/NFC: turn off
Wireless & networks/More…/Tethering & portable hotspot: setup Wi-Fi hotspot name and password
Display & lights/Cast screen: turn off
Location/Google Location Reporting: turn off for all Google accounts
Security/Owner info: set to something sensible
Security/Set up SIM card lock: lock SIM card
Security/Make passwords visible: uncheck
Privacy/Privacy Guard: check enabled by default, show built-in apps, enable guard on all except Dialer, long tap to enable location on Maps and Google Play services
Privacy/CyanogenMod statistics: uncheck enable reporting
Languages & input: uncheck Google voice typing
Languages & input/Voice Search: turn off Ok Google hotword detection, turn off offline speech recognition automatic updates, turn off personalized recognition
Backup & reset: enable backup my data, uncheck automatic restore
Superuser/Settings: set superuser access for apps only, set PIN protection

Also check out Google Settings app:

Search & Now/Google Now: turn off
Search & Now/Accounts & privacy: turn off commute sharing, search on google.com
Search & Now/Accounts & privacy: turn off Web History, turn off Personal results
Ads: reset advertising ID, opt out of interest-based ads
Android Device Manager: decide whether to allow remotely locating this device or locking and erasing data

Finally it is time to connect your device to the internet using Wi-Fi and add your Google and other accounts (carefully specify what data they should sync).

Locking

Once you tested that everything works and are satisfied with the operating system, it makes sense to again lock the bootloader. This step secures your data as everything will be wiped clean the next time anyone unlocks it again.

Reboot into the bootloader using USB (alternatively volume up + volume down + power) and lock the bootloader:

$ adb reboot bootloader
$ fastboot oem lock

Other

Screenshots

Screenshot of desktop

Encrypted partition in Debian 7

2014-08-07T00:00:00+02:00

Creating a LUKS encrypted partition with dm-crypt on Debian 7 or similar (such as Ubuntu or Raspbian) is simple. Mounting and using it in KDE is even simpler.

Preparation

Requirements

cryptsetup tool (1.4)
dm-crypt support in kernel (by default)
root or sudo permissions (run below commands as root)
empty partition

$ aptitude install cryptsetup

Determine right partition

Make sure that the target partition or block device is empty, has the right size, and you know its exact device name (eg. /dev/sdb1). All data on it will be lost therefore double check its device name using commands such as:

$ fdisk -l /dev/sdb
$ gdisk -l /dev/sdb

Overwrite with random data

To actually remove all previous data from a partition, deleting all files will not be enough as forensics can reconstruct them, therefore one has to overwrite every single byte of the partition. Overwriting them with random data also makes it impossible to distinguish which sectors include encrypted data and which not.

An obvious way of doing this with the /dev/urandom random generator is also very very slow and takes days on larger partitions (>10GB):

$ dd if=/dev/urandom of=/dev/sdb1 bs=16M

A much faster randomization method is to use a temporary encryption layer on the partition and fill it with zeros. Encrypting the zeros with an arbitrary cipher will result in data looking like randomly generated. Choose a temporary random password for this procedure:

$ cryptsetup luksFormat /dev/sdb1

WARNING!
========
This will overwrite data on /dev/sdb1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: (random)
Verify passphrase: (random)

$ cryptsetup luksOpen /dev/sdb1 sdb1_crypt
Enter passphrase for /dev/sdb1: (random)

$ dd if=/dev/zero of=/dev/mapper/sdb1_crypt bs=16M
1464+0 records in
1463+0 records out
196369113088 bytes (196 GB) copied, 5341.39 s, 36.8 MB/s

$ cryptsetup luksClose sdb1_crypt

Formatting

The encrypted partition consists of an encryption layer, such as dm-crypt with LUKS, and a file system inside it.

LUKS encryption volume

First step of setting up a user-friendly encrypted partition is formatting it as a LUKS volume. LUKS specifies a standard secure key management system and format for disk encryption.

For encryption any cipher, key size, and hashing function supported by the kernel can be used. Unfortunately the default aes-cbc-essiv:sha256 has weaknesses and it is therefore recommended to select a stronger scheme such as aes-xts-plain64 (safe for >1TB, doesn’t dependent on essiv, better tampering protection). To format the LUKS volume:

$ cryptsetup luksFormat --cipher aes-xts-plain64 /dev/sdb1

WARNING!
========
This will overwrite data on /dev/sdb1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: [password]
Verify passphrase: [password]

Because there is nothing you can do if you forget the above password, it is highly recommended to setup an alternative emergency-restore password and put it in a safe place. Luckily LUKS supports up to 8 different passwords or key slots for unlocking the partition (protected against dictionary attacks using PBKDF2 iteration scheme).

$ cryptsetup luksAddKey /dev/sdb1 --key-slot 7
Enter any passphrase: [password]
Enter new passphrase for key slot: [emergency]
Verify passphrase: [emergency]

Check LUKS header information if everything went well. It should look something like:

$ cryptsetup luksDump /dev/sdb1
LUKS header information for /dev/sdb1

Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha1
Payload offset: 4096
MK bits:        256
MK digest:      21 34 ...
MK salt:        5a e6 ...
MK iterations:  30000
UUID:           a84f890c-...

Key Slot 0: ENABLED
        Iterations:             200000
        Salt:                   81 b0 ...
        Key material offset:    8
        AF stripes:             4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: ENABLED
        Iterations:             200000
        Salt:                   33 b7 ...
        Key material offset:    1800
        AF stripes:             4000

File system inside

After setting up the encryption layer that can be opened when needed, one should format a file system inside the LUKS volume. Nowadays ext4 or btrfs are commonly chosen:

$ cryptsetup luksOpen /dev/sdb1 sdb1_crypt
Enter passphrase for /dev/sdb1: [password]

$ mkfs -t ext4 -L sdb1_crypt /dev/mapper/sdb1_crypt
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=sdb1_crypt
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
11993088 inodes, 47941678 blocks
2397083 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
1464 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, ...

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

$ cryptsetup luksClose sdb1_crypt

Usage

Now the encrypted partition is ready to be mounted and used.

If you are using KDE, the default file manager Dolphin is capable of mounting the encrypted partition just by clicking on it. Afterwards it can be used just like a normal folder, yet your data will be seamlessly encrypted in the background.

From command line it can be mounted to /mnt/sdb1 using:

$ cryptsetup luksOpen /dev/sdb1 sdb1_crypt
$ mount /dev/mapper/sdb1_crypt /mnt/sdb1

It is also possible to setup automatic mounting at boot time by specifying it in /etc/crypttab and /etc/fstab. Afterwards do not forget to update the boot-time initramfs archive.

$ grep sdb1_crypt /etc/crypttab
sdb1_crypt UUID=a84f890c-... none luks
$ grep sdb1_crypt /etc/fstab
/dev/mapper/sdb1_crypt  /mnt/sdb1  ext4  relatime  0  2
$ update-initramfs -u
$ reboot

http://www.markus-gattol.name/ws/dm-crypt_luks.html

Projectors with Android in 2014

2014-06-20T00:00:00+02:00

With the advent of new technologies in 2014, like high resolution TFT LCD displays, bright long-lasting LED lamps, and powerful systems-on-a-chip running Android 4.2 (or similar), reasonably-priced home theater projectors have finally arrived. Five or more years ago good projectors were very loud, expensive (thousands of euros), had lower resolutions (~800x600), but pretty bright light bulbs. In the mean time a series of ultra-cheap mobile mini-projectors appeared that had terribly low resolution (~320x240) and even lower brightness (~20 lumens).

Broad range of offers and features might confuse buyers and result in purchasing an over-priced and less-suitable projector for their needs. Below we will list the most important features to watch for when buying a new cheap projector with Android 4.2 and a list of examples in online shops.

Target usage scenario is watching movies and reading papers on a projection screen or ceiling in a partially dark room. For this a high enough resolution with high uniformity and a bright lamp is needed. A big issue when watching movies is also the fan noise from projectors, so it has to be minimized. One essential way of using the projector is by connecting it to a computer (or other multimedia devices) using HDMI or VGA cables. On the other hand the built-in powerful mini-computer with Android enables streaming of movies or similar content directly from USB storage devices, NAS servers, or even the internet (like YouTube, PopcornTime). To avoid wireless network performance issues it is a good idea to connect the projector with an Ethernet cable or have support for larger SD cards or USB disks to temporary store content.

Important features

Minimal projector features to watch for:

Native resolution: 1280x768
Lamp: 200 W LED lamp (with 50000 hours life span)
Brightness: 4500 ANSI lumens
Contrast ratio: 4000:1 (dynamic)
Uniformity: 90%
Noise: 30 dB (low noise)

Minimal system features to watch for:

OS: Android 4.2
CPU: ARM Cortex A9 dual-core 1.5 GHz
RAM: 1 GB
Internal storage: 4 GB

Interfaces:

2x USB port
2x HDMI port
VGA port
Audio L/R out
SD card slot (up to 32 GB)
RJ45 port (Ethernet)
WiFi 802.11 b/g/n

Other:

DLNA/UPnP support
Remote controller
Projection distance: 1.5m-5.5m
Projection methods: front, rear, ceiling, rear ceiling
Power consumption: 200 W

Online shops

Quick comparison of cheap home theater projectors with good price-performance ratio available in online shops. Only missing or noteworthy things are exposed, although it is questionable how to evaluate misleading specifications on similar products.

Seem good:

ACME 86+WIFI, ACME New 86+, ACME LED86+
- unknown WiFi?
- price: US$ 355.00 at AliExpress
- price: US$ 394.15 at AliExpress
KT 86+W
- unknown uniformity?
- noise 32 dB
- only WiFi 802.11 b/g
- price: US$ 349.00 at AliExpress
Smart 86, SEESMART LED86
- unknown uniformity, noise?
- only WiFi 802.11 b/g
- price: US$ 356.00 at AliExpress
- price: US$ 356.15 at AliExpress
SmartIdea LED-86+(Wifi)
- unknown uniformity, noise?
- only WiFi 802.11 b/g
- price: US$ 355.00 at AliExpress

Seem poor:

ATCO CT03H2 New
- unknown uniformity, noise?
- unknown RAM?
- no RJ45 port
- price: US$ 360.36 at AliExpress
DroidBeam
- no RJ45 port
- brightness only 3000 lumens, contrast 2000:1
- price: US$ 408.56 at Monastaraki
EJL EPW58D
- unknown CPU, RAM?
- brightness only 3000 lumens, contrast 1000:1, uniformity 80%
- only Android 4.0.4
- only WiFi 802.11 b/g
- price: US$ 380.47 at LinkDelight
- price: US$ 451.20 at DX
Oley H2
- unknown uniformity, noise?
- brightness only 3000 lumens, contrast 2000:1
- only Android 4.1
- no RJ45 port
- price: US$ 440.12 at DX
RuiQ SV-128
- unknown uniformity, noise?
- unknown CPU, RAM, internal storage?
- brightness only 2600 lumens, contrast 1000:1
- only Android 4.0
- no WiFi
- price: US$ 383.71 at DX

http://www.alectrosystems.com/video/projection/projector_specs.htm

[UI-part3] Incremental learning from data streams

2014-05-21T00:00:00+02:00

Course: https://ucilnica.fri.uni-lj.si/course/view.php?id=81
Lecturer: izr. prof. dr. Zoran Bosnić
Language: English
Date: 2014-05-14

Course overview (part 3):

comparison of stationary learning vs. incremental learning; streams, design, limitations
data summarization: statistics, histogram, wavelets, condensing datasets
incremental learning models: incremental decision trees (VFDT, functional tree leaves), Hoeffding bound
concept drift/novelty detection, reacting to changes: extreme values, decision structure, frequency, distances
clustering from data streams: hierarchical, micro, grid clustering,
evaluation of streaming algorithms
matrix factorization with temporal dynamics (recommendation systems)

Lectures with hands-on exercises in Statistical package R. Homework problem for grading: competition in modeling given streaming data. Project: will be discussed individually, taking some data related to PhD and doing something incrementally.

Learning from data streams

Date: 2014-05-14

Introduction

Static learning:

fixed dataset
relational/attributional form
classification/regression prediction problems

Incremental learning:

data streams = changing data
concept drift = unpredictable data distribution
time series = measurements with timestamps, transformation into relational data is possible
sliding window = holds most recent examples
- sequence-based window – 20 last measurements
- timestamp-based window:
  - linear – 20 samples with 15 min between
  - non-linear – 20 samples with increasing space between them
  - adaptive – shrinks on demand (for concept drift)

Algorithm: ADWIN (ADaptive sliding WINdow)

split window into two subwindows such that their average is the same (test all combinations)
drop elements if it is not
comparison of means from two subwindows must take into account their stability (from how many elements the mean was computed)
one parameter $\delta$

Data synopsis

Compress incoming information from a stream to make processing feasible, not store all data.

representative sample: reservoir sampling algorithm
histograms
sketches
discrete transforms and wavelets: Haar wavelets

Predicting from data streams

Date: 2014-05-21

Options:

for time series: statistics, transformations, fit coefficients
for relational/attributional data: adapt common learning algorithms, specialized incremental learning algorithms
transform time series into attributional form:
- temporal attributes – periodical attributes based on assumptions what makes sense (eg. daily/weekly sin/cos function)
- historical attributes – eg. measurements at some hours before
- real values – for manual checking

Incremental models do not require all historical data to be available and can be build incrementally. Incremental learning slides a window through data and updates the model with new examples as they arrive.

Algorithm: Very fast decision tree algorithms (VFDT)

only discrete attributes
tree is not binary
split a leaf node only if there is sufficient statistical evidence

Growing the tree:

update statistics in the leaf
evaluate $H()$ on all attributes, compute difference between best two attributes: \[ \Delta H = H(a_1) - H(a_2) \]
compare Hoeffding bound (Hoeffding, 1963): \[ \epsilon = \sqrt{\frac{R^2 ln(2 / \delta)}{2n}} \]

Concept drift detection

Distribution of examples or what is being modeled can change over time – eg. as a consequence of a hidden variable or process. We must be cautious to differentiate from temporal noise.

Categorizations based on:

data management:
- full memory (by increasing recent weights)
- partial memory (fixed or adaptive window)
detection methods:
- monitor performance indicators
- compare time-windows
adaptation capability:
- blind methods (restart at regulated interval)
- informed methods

Tests:

Page-Hinkley test (PH test): compute difference against average, alarm if larger than allowed
Statistical process control: compute probability of error $p_i$ and its standard deviation $s_i$, system switches between states according to bounds (“in control”, “warning”, “out of control”)

Clustering from data streams

Date: 2014-05-28

online clustering
evaluating incremental algorithm

In online clustering the task is to maintain a continuously consistent good clustering using a small amount of memory and time (grouping objects into groups, unsupervised learning).

Approaches:

hierarchical clustering: build micro clusters for cluster features (CF), maintain statistics for each one (not storing examples), later combine them into macro clusters
partitioning clustering (k-means)
density-based clustering
grid-based clustering
model-based clustering

Algorithm: BIRCH

Hierarchical clustering approach with acronym meaning Balanced Iterative Reducing and Clustering using Hierarchies (Zhang et al., 1996).

build a tree
every node has 6 cluster features and descendant subtrees
each node summarizes the statistics of descending nodes
parameters: $B$ - branch factor, $T$ - maximum absorption distance
put new examples arrive, the closest CF can absorb it if closer than radius $T$, if not a new CF entry is added, if there is no room split the parent node

An extension is CluStream (Aggarwal et al., 2003) that maintains just $q$ micro clusters, discards oldest.

Evaluation of streaming algorithms

Performance measures: MSE, CA…

Approach in batch learning is to split data into training set (training set, validation set) and test set.

Issues in streaming data:

dataset not fixed
concept drifts
decision models change over time

Approaches influenced by order of examples:

holdout and independent test set
predictive sequential (prequential) evaluation

Solution is to use prequential estimate with forgetting mechanism (using either time window or fading factors).

Q-estimate is used to compare two algorithms. Taking a logarithm of accumulated losses provides positive values if first algorithm is better than the second.

\[ Q_i(A,B) = log(\frac{S^A_i}{S^B_i}) \]

Sampling promotes community structure in social and information networks

2015-07-01T00:00:00+02:00

Scientific paper

N. Blagus, L. Šubelj, G. Weiss, and M. Bajec, “Sampling promotes community structure in social and information networks,” Phys. A Stat. Mech. its Appl., p. 15, 2015.

Abstract

Any network studied in the literature is inevitably just a sampled representative of its real-world analogue. Additionally, network sampling is lately often applied to large networks to allow for their faster and more efficient analysis. Nevertheless, the changes in network structure introduced by sampling are still far from understood. In this paper, we study the presence of characteristic groups of nodes in sampled social and information networks. We consider different network sampling techniques including random node and link selection, network exploration and expansion. We first observe that the structure of social networks reveals densely linked groups like communities, while the structure of information networks is better described by modules of structurally equivalent nodes. However, despite these notable differences, the structure of sampled networks exhibits stronger characterization by community-like groups than the original networks, irrespective of their type and consistently across various sampling techniques. Hence, rich community structure commonly observed in social and information networks is to some extent merely an artifact of sampling.

[AA-part3] CUDA architecture

2014-05-26T00:00:00+02:00

Course: https://ucilnica.fri.uni-lj.si/course/view.php?id=89
Lecturer: doc. dr. Tomaž Dobravec
Language: Slovenian
Date: 2014-05-12

Course overview (part 3):

hardware and programming aspects of CUDA architecture
solving standard problems and a comparison between serial (CPU) and parallel (GPU) implementation
basics of the OpenCL programming environment
CUDA architecture in the context of the OpenCL environment

Učili se bomo CUDA. Seminarska naloga bo napisati program v Java in CUDA, lahko pa tudi Python in OpenCL. Rok in zagovor seminarskih naj bodo 9.6. po pedagoški delavnici.

Project idea: given Java skeleton for an image processing application, implement Seam Carving using CUDA. Otherwise propose your project.

Arhitektura CUDA

Razno:

host – CPU
device – GPU
SM – streaming multi-processor, ima svoj cache, izvaj isto kodo
celo v skupkih izvajajo isto kodo
SP – posamezen procesor
kernel/ščepec – osnovna komponenta kode
thread/nit – izvajajo kernele, potrebno podati kako se naj izvajajo
1 blok niti se vedno izvrši na istem SM
razporedi niti v bloke po največ 512 niti, ki med seboj lahko komunicirajo, z ostalimi pa ne morejo sodelovati (saj se lahko izvajajo hkrati)
programer pripravi mrezo blokov, ki se potem izvaja, na koncu veš, da se bodo vsi bloki izvršili
pomnilnik se ne briše med različnimi programi

Date: 2014-05-19

Recommended tool is Nsight (CUDA Eclipse-like IDE):

always check for returned errors
files *.cu

Memory model:

registri (32-bitni) so na vsak blok (8192), nato se razporedijo med threadi, največ 16 na en ščepec (kernel) pri polni obremenitvi, če compiler ne uspe, ga da v lokalni pomnilnik
lokalen pomnilnik se uporablja znotraj ščepca, počasen del DRAMa, skupaj 8kB
shared memory uporabljajo vse niti istega ščepca, če podatek več kot enkrat potrebuješ, se ga splača prekopirati sem, to je ključ hitrega programa, hiter
global (device) memory, lahko uporablja za komunikacijo med ščepci, počasen DRAM, skupaj par GB
host (CPU) memory

Kernel grid:

dokler ne konča izvajanja enega kernela, ne gre na drugega
lightweight context switching med bloki niti, zato lahko več blokov hkrati naloži
bistvo veliko dobro obteženih blokov
velikost bloka >=32, saj sicer izvajanje v snopih (warp) manj učinkovito

Java native interface

Date: 2014-05-26

V Javi ni možno dostopati do posebnih sistemskih klicev kot je CUDA, je pa možno z JNI poklicati C knjižnico, ki nato kliče naprej.

// -Djava.library.path=...
static {
    System.loadLibrary("JNIFirst");
}
private native static int sestej(int a, int b);

cd src
javah -jni JNI
# (generates JNI.h from JNI.java)
# (prepare JNI.c)
gcc -I$JNI_INCLUDE -c JNI.c -o JNI.o
gcc -dynamiclib -o libJNIFirst.jnilib JNI.o

Concept: Shift

How to shift all elements of a vector for 1 element to the left using CUDA.

Note: Avoid simultaneous read and write and synchronize execution using __syncthreads() (must be outside a conditional clause).

__global__ void shiftLeft(int *a) {
    __shared__ int mem[N];  // for all blocks
    int idx = threadIdx.x;
    mem[idx] = a[idx];
    __syncthreads();  // must be outside conditional clauses
    if(idx < N - 1)
        a[idx] = mem[idx + 1];
}

Concept: Reduction

How to sum all elements of a vector using CUDA. Parallel operation is called reduction.

__global__ void reduce(int *a) {
    int i = threadIdx.x;
    for(int stride=1; stride < N; stride *= 2) {
        if(i % stride == 0)
            a[2*i] += a[2 * i + stride];
        __syncthreads();
    }
}

Note: Using shared memory instead of global would speed it up.

Note: In each iteration we create holes in the vector that still use resources in a warp. By compacting and leaving holes out we can improve this.

__global__ void reduce(int *a) {
    int i = threadIdx.x;
    extern __shared__ int mem[];
    mem[i] = a[2 * i] + a[2 * i + 1];
    for(int stride=N/4; stride >= 1; stride /= 2) {
        __syncthreads();
        if(i < stride)
            mem[i] += mem[i + stride];
    }
    __syncthreads();
    a[0] = mem[0];
}

Concept: Histogram

How to count number of occurences of characters in a string (i.e. histogram).

Note: Problem is simultaneous read and write for a character. Solution is a special CUDA atomic operation.

Note: Neighbouring threads should access neighbouring memory locations, because DRAM works faster when accessed in blocks.

__global__ void histogram(unsigned char *str, int len, unsigned int *hist) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    int stride = blockDim.x * gridDim.x;
    while(i < len) {
        //hist[niz[i] - 'a']++;  // wrong
        atomicAdd(&hist[niz[i] - 'a'], 1);
        i += stride;
    }
}

Oddaja projekta (izvorna koda, kratko poročilo) in prezentacija 16.6.2014 ob 14:00.

[UI-part2] Inductive logic programming

2014-05-07T00:00:00+02:00

Course: https://ucilnica.fri.uni-lj.si/course/view.php?id=81
Lecturer: akad. prof. dr. Ivan Bratko
Language: English
Date: 2014-04-02

Course overview (part 2):

Inductive Logic Programming (ILP) is an approach to machine learning in which learned descriptions are represented in logic, typically in predicate logic.This is the most expressive hypothesis language among all the approaches to machine learning. ILP also makes use of “background knowledge”, that is knowledge that is known to the learning program before learning begins. Some basic methods of ILP will be discussed.

ILP Programming

See presentation files for all lectures.

[AA-part2] Parallel algorithms

2014-05-05T00:00:00+02:00

Course: https://ucilnica.fri.uni-lj.si/course/view.php?id=89
Lecturer: prof. dr. Borut Robič
Language: Slovenian
Date: 2014-03-24

Course overview (part 2):

models of parallel computation: PRAM and versions CRCW, CREW, EREW
design and performance of parallel algorithms: simulation of models, Brent theorem
performance guarantees of CRCW, CREW, EREW simulation: O(log n)-time sorting
class NC: efficiently parallel solvable problems, robustness of NC, is P = NC?

Samo predavanja, ustni izpit (snov iz predavanj in seminarske), a ne bo pisnega izpita ali domačih nalog. Pogledali bomo osnovne pojme iz področja modelov računanja in algoritmov. Ustni izpit kadar zelite (lahko isti dan kot Kodek), gremo skozi snov, intuitivno kar vse to pomeni.

Homework: Skupinska seminarska naloga

Poglej za vsakega od modelov računanja (tudi bolj natančne) kaj je bistveno. Do konca naših predavanj.

prednosti
slabosti
razmišljanje o konkretnem problemu in kateri je primeren zanj
koliko me stane, ce preidemo iz enega v drugega
cel svet problemskih prostorov (kot pri zaporednih, npr. nedeterminističnost, aproksimacijski)
katere topologije so primerne za kateri problem, kako aktivnosti v modelu preslikati na dejansko arhitekturo, mapping algoritma na stroj, da ključne komunikacije ne gredo po najdaljših povezavah
pri prijavi teme se nekdo veliko s tem ukvarjal

Uvod v modele računanja

Model računanja je abstrakcija, običajno formalno zapisana, ki zavzema vse kaj je bistvenega za algoritem, porabljen čas ali drug vir.

Modeli:

PRAM
Turingov stroj
Lambda račun
RAM (opisuje Von-Neumannovo arhitekturo)
Rekurzivne funkcije
Postov stroj
Registrski stroj
DNK računanje

Turingov stroj

Turingov stroj uporabimo, da ugotovimo, kateri računski problemi so rešljivi in kateri ne. Za nerešljive probleme ne obstaja noben algoritem.

Kateri računski problemi so rešljivi? S tem se ukvarja teorija izračunljivosti (computability theory).

Ali imajo diofantske enačbe ali sistem enačb rešitev v celih številih? Izkaže se, da algoritem ne obstaja (1972), ki bi znal za katerokoli diofantsko enačbo izpisati ali je rešljiva. Teorija izračunljivosti nam pomaga, da se ne lotevamo nerešljivih problemov.

Kakšen čas/prostor je potreben za rešitev rešljivega računskega problema? S tem se ukvarja teorija računske zahtevnosti.

Na primeru urejanja števil lahko ob ustrezni analizi algoritmov odkrijemo, da ni rešljivo hitreje kot v $O(n \log n)$ časa.

Turingov stroj je model zaporednega računanja.

Model PRAM

Kaj pa vzporedno (paralelno) računanje? Pri takem računanju sodeluje več procesorjev, ki so sposobnih hkrati reševati en problem in pri tem sodelujejo.

Glavna težava pri modeliranju takega računanja je bila kako čim bolj verno modelirati čas, ki se porablja za komunikacijo med procesorji. Posledično je nastalo več predlogov za model vzporednega računanja.

Eden prvih modelov je bil PRAM (parallel random access machine):

čas potreben za komunikacijo zanemarjamo
rezultati bodo zato optimistični, torej predstavljajo spodnjo mejo za časovno zahtevnost vzp. rač.

[skupni  pomnilnik]    -- potencialno neskončen
 ^     ^         ^
 |     |         |
 v     v         v
[P1]  [P2]  ... [Pn]   -- končno število procesorjev

Model PRAM

PRAM (parallel random access machine) ima množico procesorjev $P_1, P_2, ..., P_n$, ki so povezani s skupnim pomnilnikom sestavljenim iz celic. Pri modeliranju detajle za katere ocenimo, da niso pomembni, ignoriramo.

Model skupnega pomnilnika (značilnosti/idealizacije):

potencialno neskončen (lahko ga povečaš za koliko želiš, a vsakič končen)
sestavljen iz pomnilniških celic
velikosti celic so enake (npr. vse 32- ali 64-bitne), toda potencialno neskončne
vsaka celica je dostopna vsakemu procesorju $P_i$
dostopni čas enak za vsako celico in vsak procesor $P_i$ (=en korak/enota časa)

Model procesorjev:

potencialno neskončno (npr. $n, n^2, \sqrt{n}$ procesorjev)
vsi procesorji $P_i$ izvajajo isti algoritem/program (čeprav bo kdaj kateri delal tudi kaj drugega)
vsi lahko dostopajo do vseh pomnilniških celic v enoti časa

Sočasno dostopanje

Kaj se zgodi, če procesorji sočasno želijo dostopati do iste celice? Odvisno od operacije, ki jo želijo izvesti na celici (branje ali vpis).

sočasno branje: načeloma ni težav v modelu (vsebino celice dobijo vsi v naslednjem koraku)
sočasen vpis: če vsaj dva želita vpisati različno vsebino, ni jasno čigava vsebina bo obveljala (problem!)

Glede na to ločimo več različic PRAMa:

EREW PRAM (exclusive-read, exclusive-write):

v vsakem koraku: iz iste celice lahko bere kvečjemu 1
v vsakem koraku: v isto celico lahko vpiše kvečjemu 1
da je to izpolnjeno mora zagotoviti razvijalec algoritma/programa
model je najbližje realnosti (še najmanj dela za realizacijo)

CREW PRAM (concurrent-read, exclusive-write):

v vsakem koraku: iz iste celice lahko bere več kot 1 procesor
v vsakem koraku: v isto celico lahko vpiše kvečjemu 1 procesor
za to mora zagotoviti razvijalec

CRCW PRAM (concurrent-read, concurrent-write):

v vsakem koraku: iz iste celice lahko bere več kot 1 procesor
v vsakem koraku: v isto celico lahko vpiše več kot 1 procesor (o tem kaj se bo zgodilo se je potrebno dogovoriti)

Kateri procesor bo zmagal pri sočasnem vpisu pri CRCW PRAM? To določa t.i. način sočasnega vpisa (mode) pri CRCW:

skladni vpis (consistent mode):
- vsi procesorji morajo vpisati enak podatek v isto celico
- za to mora poskrbeti razvijalec algoritma
poljuben vpis (arbitrary mode):
- zmagovalec je naključen (za nas)
- to bo moral upoštevati razvijalec (nedeterministično obnašanje)
prednostni vpis (priority mode):
- zmaga procesor z najvišjo prioriteto (npr. z najnižjim indeksom)
- to bo moral upoštevati razvijalec
združeni vpis (fusion mode):
- nad podatki se izvede operacija, njen rezultat pa se vpiše v celico
- operacija (z dvema operandoma) mora biti komutativna in asociativna
- npr. disjunkcija, konjunkcija, max, min, množenje, seštevanje (odštevanje ni komutativno)

Vzporedni algoritmi

Date: 2014-03-31

Tekom predavanj želimo pokazati, da je vseeno katerega od modelov izberemo, saj lahko le-ti drug drugega simulirajo. Cena za simulacijo je logaritemska. Pri vzporednem računanju nas zanimajo poli-logaritmični algoritmi $\log^p(n)$, ne polinomski $n^k$.

Tehnika: Preusmerjanje kazalcev (pointer jumping)

Preusmerjanje kazalcev je tehnika/metoda za razvoj vzporednih algoritmov. Posplošitev metode “deli in vladaj” iz razvoja zaporednih algoritmov.

Primer: Rangiranje seznamov (list ranking)

Def. (rangiranje seznama): Dan je seznam L z vozlišči (linked list o->o->o->...->o, vsak vsebuje vrednost $d[i]$ in kazalec $next[i]$). Ne vemo koliko je vozlišč in premikamo se lahko le na naslednjika. Za vsako vozlišče seznama L izračunaj rang (=razdalja tega vozlišča do konca seznama). Razdalja zadnjega do konca je $0$. Npr.: L->[3]->[2]->[1]->[0]

Zaporedni alg.:

ideja: začnemo na začetku seznama, se sprehodimo do konca in nazajgrede štejemo rang
potrebna dva sprehoda od začetka do konca seznama
$2n$ obiskov vozlišč => asimptotična časovna zahtevnost $\Theta(n)$

Ali obstaja vzporedni alg. za PRAM, ki ima (asimptotično) časovno zahtevnost manjšo od $\Theta(n)$ (npr. $\Theta(\sqrt(n)))$, $\Theta(\log^p(n)))$)? Da. Obstaja EREW PRAM z $n$ procesorji in ima časovno zahtevnost $O(log(n))$.

Vzporedni alg.:

seznam je končen in sistem dovolj velik
vsakemu vozlišču priredimo lastno PE
ideja: v vsakem koraku alg. se naj seznam razdeli na 2 podseznama, pri tem upoštevamo učinek trganja na vrednosti $d$ (trganje je preusmerjanje kazalcev)
najprej vsako vozlišče dobi vrednost 1, zadnje 0
lastnost vozlišč:

\[ d[i] = \begin{cases} 0 & \text{if } next[i] = nil \\ 1 + d[next[i]] & \text{else} \end{cases} \]

RankComputation(L):
  forall i inparallel do
    if next[i] = nil then d[i] <- 0 else d[i] <- 1
  while obstaja voišče i: next[i] != nil do
    forall i inparallel do
      if next[i] != nil then
        d[i] <- d[i] + d[next[i]]
        next[i] <- next[next[i]]

Opombe:

forall i inparallel do stavek izvedejo paralelno hkrati vsi tisti procesorji katerih vozlišča so omenjena
prevajalnik mora poskrbeti, da se branja izvedejo pred vpisi
- iz: forall i inparallel do A[i] <- B[i]
- v: forall i inparallel do temp[i] <- B[i]; forall i inparallel do A[i] <- temp[i]
- problem nastane pri: forall i inparallel do A[i] <- A[i-1]
- za to poskrbi prevajalnik
pogoj while obstaja voišče i se da preveriti na:
- CRCW PRAM z združenim vpisom v konstantnem času $O(1)$
- EREW PRAM pa je potreben čas $O(log(n))$

Primer: Računanje predpon (prefix computation)

Tudi tu se uporablja preusmerjanje kazalcev.

Def. (računanje predpon): Dano je zaporedje $x_1,...x_n$ in binarna asociativna operacija $\otimes$. Izračunati je potrebno vrednosti $y_1,...y_n$, kjer je $y_1 = x_1$, $y_{i} = y_{i-1} \otimes x_i$ za $i >= 2$.

Npr.: $y_1 = x_1$, $y_2 = x_1 \otimes x_2$, $y_3 = x_1 \otimes x_2 \otimes x_3, ...$

Zaporedni alg.:

zahteva $O(n)$ časa

Se da na vzp. način hitreje? Da. V času $O(log(n))$ celo na EREW PRAM z $n$ PE.

Vzporedni alg.:

vhodne podatke $x_1,...x_n$ shranimo v seznamu L (linked list)
metoda: preusmerjanje kazalcev

PrefixComputation(L):
  forall i inparallel do
    y[i] <- x[i]
  while obstaja vozlišče i: next[i] != nil do
    forall i inparallel do
      if next[i] != nil then
        y[next[i]] <- y[i] * y[next[i]]
        next[i] <- next[next[i]]

Opomba:

pogoj while obstaja voišče i se da preveriti na:
- CRCW PRAM v $O(1)$
- EREW PRAM (z $n$ PE) pa v $O(log(n))$

Problem uporabe računanja predpon: Dano je dvojiško drevo z $n$ vozlišči (koren na globini $0$). Izračunaj globino vsakega vozlišča tega drevesa.

Z zaporednim alg. v $O(n)$ kateri od obhodov drevesa (depth-first, breadth-first).
Z vzporednim alg. na EREW PRAM (z $n$ PE) se da problem rešiti v času $O(log(n))$, tudi za izrojena drevesa. Se prevede na problem računanja rangov.

Ocenjevanje zmogljivosti vzporednih algoritmov

Date: 2014-04-07

Zanimala nas bo cena, delo, pohitritev in učinkovitost vzporednih algoritmov.

Naj bo dan problem $P$ in njegov primerek velikosti $n$ (npr. uredi $n$ števil).

Def.: $T_{seq}(n)$ označuje časovno zahtevnost najboljšega zaporednega algoritma za reševanje problema $P$.

Naj bo dan vzporedni algoritem $A$ za reševanje problema $P$.

Def.: $T_{par}(p, n)$ označuje časovno zahtevnost izvajanja algoritma $A$ za reševanje problema $P$ na $p$ procesorjih v PRAM modelu.

Def.: Cena (ang. cost) vzporednega algoritma $A$ je $C_p(n) = p \cdot T_{par}(p, n)$. Celoten porabljan čas, tako za računanje kot čakanje.

Def.: Delo (ang. work) vzporednega algoritma $A$ je $W_p(n)$. To je število vseh operacij, ki so jih dejansko opravile vse procesne enote pri izvajanju algoritma $A$.

\[ W_p(n) \leq C_p(n) \]

Cena bo minimalna, kadar vsi procesorji končajo hkrati (in vsi delajo hkrati).

Def.: Pohitritev (ang. speedup) vzporednega algoritma $A$ je $S_p(n) = \frac{T_{seq}(n)}{T_{par}(p, n)}$.

Def.: Učinkovitost (ang. efficiency) vzporednega algoritma $A$ je $E_p(n) = \frac{S_p(n)}{p}$ (koliko pohitritve nam je v povprečju prinesel posamezen procesor). Če vstavimo formulo za pohitritev, dobimo: $E_p(n) = \frac{T_{seq}(n)}{C_p(n)}$.

Velja: $S_p(n) \leq p$ (algoritem bo tekel največ $p$-krat hitreje), $E_p(n) \leq 1$ (učinkovitost bo največ $1$).

Izrek o enostavni simulaciji

Govori o izvajanju algoritma na manjšem vzporednem računalniku: Delovanje PRAMa lahko simuliramo z manjšim PRAMom (vzporedni algoritem lahko izvedemo tudi na manjšem vzporednem računalniku). Pri tem dosežemo zmanjšanje zmogljivosti algoritma, a poslabšanje je navzgor omejeno.

Izrek: Naj bo $A$ vzporedni algoritem, ki se izvaja na PRAMu s $p$ procesorji $t$ časa. Manjši PRAM enakega tipa, ki ima $p' < p$ procesorjev, lahko izvede algoritem $A$ v času $O(\frac{t \cdot p}{p'})$. Cena algoritma $A$ na manjšem PRAMu pa je kvečjemu dvakrat večja od cene na večjem PRAMu:

\[ C_{p'} \leq 2C_p \]

Dokaz: Vsak korak algoritma $A$ lahko manjši PRAM izvede kvečjemu v $\lceil\frac{p}{p'}\rceil$ časa.

Manjši PRAM izvede cel algoritem $A$ v času $t' = t \lceil\frac{p}{p'}\rceil = O(\frac{t \cdot p}{p'})$.

Cena: $C_{p'} = p' \cdot t' = p' \cdot t \lceil\frac{p}{p'}\rceil \leq p' \cdot t \cdot (\frac{p}{p'} + 1) = tp + tp' \leq tp + tp = 2tp = 2C_p$ (manj procesorjev, večji čas).

Vprašanje: Kaj, če je $p' = 1$? To je še PRAM, toda z enim procesorjem, kar pa je že model RAM za zaporedno računanje. Iz izreka sledi (ker je $p' = 1$):

Algoritem $A$ lahko simuliramo na zaporednem modelu v času $O(tp)$.
Zaradi izreka je $C_1 \leq 2C_p$ in po drugi strani je $C_1 = 1 \cdot T_{par}(1, n) \geq T_{seq}(n)$, zato: $\frac{1}{2}T_{seq}(n) \leq C_p(n)$.
Iz tega sledi: $C_p(n) = \Omega(T_{seq}(n))$ (cena vzporednega algoritma je navzdol omejena z najboljšo časovno zahtevnostjo zaporednega algoritma).

Denimo, da je $C_p(n) = \Theta(T_{seq}(n))$. Ker je $C_p(n)$ po definiciji enaka $p \cdot T_{par}(p,n)$, sledi da je: $p \cdot T_{par}(p,n) = \Theta(T_{seq}(n))$. Torej je: $T_{par}(p,n) = \Theta(\frac{T_{seq}(n)}{p})$. Algoritem $A$ na $p$ procesorjih bi bil $p$-krat hitrejši od najhitrejšega zaporednega algoritma. Očitno $A$ izkorišča vseh $p$ procesorjev ves čas reševanja problema in reši problem v najmanjšem možnem času.

Posledica (in definicija): Vzporedni algoritem je učinkovit, če zanj velja:

\[ C_p(n) = \Theta(T_{seq}(n)) \]

Koliko procesorjev rabimo?

\[ p = \Theta(\frac{T_{seq}(n)}{T_{par}(p, n)}) = \Theta(S_p(n)) \]

Velja tudi $S_p(n) = \Theta(p)$ (če obrnemo). Pri učinkovitem vzporednem algoritmu je pohitritev približno $p$-kratna.

Diskusija

Intuicija: Več procesorjev reši problem v krajšem času.
Teorija: Če je vzporedni algoritem učinkovit, potem $p$ procesorjev reši problem $p$-krat hitreje kot najboljši zaporedni algoritem.
Praksa: Izkaže se, da ko $p$ preseže neko mejno vrednost (ta je odvisna od problema in algoritma), potem $S_p(n)$ ni več linearna funkcija od $p$. Lahko dodajamo procesorje, vendar pohitritev ne bo linearna, zato učinkovitost pada (vsak procesor prispeva k pospešku manj, kot bi lahko).

Kaj je vzrok za upad pohitritve? Običajno je vzrok komunikacija med procesorji.

Brentov izrek

Vzpostavlja povezavo med št. procesorjem in št. operacij, ki so bile izvedene.

Izrek: Naj bo $A$ vzporedni algoritem, ki se izvaja na nekem PRAMu z neomejenim št. procesorjev $t$ časa in pri tem izvede $m$ operacij. Algoritem $A$ se lahko izvede na PRAMu enakega tipa in na $p$ procesorjih v času $t'$:

\[ t' = O(\frac{m}{p} + t) \]

Dokaz: Recimo, da $A$ na prvem PRAMu v $i$-tem koraku ($i$ gre od 1 do $t$) izvede $m(i)$ operacij. Torej vseh operacij je $\sum_{i=1}^t m(i) = m$. Drugi PRAM $i$-ti korak algoritma $A$ lahko izvede v daljšem času (ker ima na voljo omejeno število procesorjev)—v $\lceil\frac{m(i)}{p}\rceil$ korakih. Vseh korakov na drugem PRAMu je zato:

\[ \sum_{i=1}^{t} \lceil\frac{m(i)}{p}\rceil \leq \sum_{i=1}^{t} ( \frac{m(i)}{p} + 1 ) = \frac{1}{p} \cdot \sum_{i=1}^{t} m(i) + \sum_{i=1}^{t} 1 = \frac{m}{p} + t \]

Uporaba: V problemu so dana števila $x_1,\ldots,x_n$, izračunaj maksimum teh števil na EREW PRAM.

EREW PRAM bi računal max z drevesnim algoritmom:

x1     x2     x3     x4    x5     x6    x7    x8
  \   /         \   /        \   /       \   /
   [ ]           [ ]          [ ]         [ ]
    \...
...
4 procesorji na 1. nivoju
2 na 2. nivoju
1 na zadnjem nivoju

Če ima PRAM na voljo vsaj $\frac{n}{2}$ procesorjev (tj. $\Theta(n)$), lahko izračuna maksimum v času $\Theta(\log{n})$.

Vprašanja:

Kaj pa če bi bilo na voljo le npr. $O(\sqrt{n})$ procesorjev, kako to vpliva na čas računanja?
Za največ koliko lahko zmanjšamo št. procesorjev (asimptotično), da se bo drevesni algoritem še vedno izvedel v času $\Theta(\log{n})$?

Uporabimo Brentov izrek:

Drevesni algoritem traja $t = \log{n}$ korakov na neomejenem PRAMu.
Število vseh operacij, ki jih izvede, je $m = n - 1$ (število notranjih vozlišč v drevesu).
Če je na voljo le $p$ procesorjev, lahko izvedejo drevesni algoritem v času $O(\frac{n-1}{p} + \log{n})$ (po Brentu).
Odgovor na 1. vprašanje: če bi bil $p=\Theta(\sqrt{n})$, potem bi bil čas algoritma $t = O(\frac{n-1}{k \cdot \sqrt{n}} + \log{n})$, kar je $O(\sqrt{n} + \log{n})$ ($k$ nesemo ven, delimo s $\sqrt{n}$, zanemarimo $\frac{1}{\sqrt{n}}$), kar je $O(\sqrt{n})$.
Odgovor na 2. vprašanje: če za $p$ vzamemo $p=\Theta(\frac{n-1}{\log{n}})$ procesorjev, potem bi drevesni algoritem tekel v času $O(\frac{n-1}{\frac{n-1}{\log{n}}} + \log{n}) = O(\log{n})$.

Npr.: Če imamo 1024 števil ($n=1024$). Na običajen način bi rabili $\frac{n}{2} = 512$ procesorjev. Lahko pa jih vzamemo 100 in bomo algoritem lahko izvedli v času, ki je primerljiv s prejšnjim.

Primerjava modelov PRAM

Date: 2014-04-14

Gre za izreke o razlikovanju modelov PRAM (ang. model separation theorems).

Vprašanje: Ali se CRCW, CREW in EREW PRAM modeli res razlikujejo po svoji “moči”? Če se, za koliko?

Motivacija: Lažje je sestaviti paralelni algoritem za CRCW kot pa za EREW. Ali je povečanje časa pri prehodu iz CRCW na EREW lahko poljubno veliko?

Kako to ugotovimo? Če se ti modeli res razlikujejo, po svoji “moči”, potem morata obstajati dva problema, $P_1$ in $P_2$, na katerih se pokaže razlika v njihovi “moči”.

\[ CRCW \supset CREW \supset EREW \]

Posamezna množica predstavlja probleme, ki jih je posamezen model sposoben rešiti v danem času (asimptotično gledano). CRCW je sposoben rešiti največ problemov v danem času, CREW malo manj, EREW pa najmanj. Domnevamo, da so ti razredi strogo vsebovani en v drugem.

Ta dva problema, ki ju iščemo, sta med EREW in CRCW. $P_1$ je v CRCW, $P_2$ pa v CREW.

$P_1$ reši CRCW v nekem času $T(n)$, v katerem ga CREW ne more
$P_2$ reši CREW v nekem času $T'(n)$, v katerem ga EREW ne more

Kako najdemo taka problema, če sploh obstajata?

Odnos CRCW-CREW (patološki problem $P_1$)

Ali obstaja problem $P_1$, ki ga z enakim št. procesorjev CRCW reši asimptotično hitreje kot katerikoli CREW?

Da. $P_1$ je problem iskanje največjega števila med danimi $n$ števili ($\mbox{max} \{A[1], A[2], \ldots, A[n]\}$).

Trditev: CRCW PRAM (z $n^2$ PE) reši $P_1$ v času $O(1)$ (če damo zadosti procesorjev, reši to v konstantnem času).

Dokaz: Konstrukcija algoritma:

IzracunajMax(A, n):
  # m[i] bo true, dokler se ne najde število, ki je večje od A[i]
  forall i in {1,...,n} inparallel do:
    m[i] := true

  # za vse različne pare števil i,j:
  #   če najdemo nek A[j], ki je večji od A[i], potem A[i] ne more biti maksimalen element
  forall i,j in {1,...,n}, i!=j inparallel do:
    if A[i] < A[j]:
      m[i] := false

  forall i in {1,...,n} inparallel do:
    if m[i] == true:
      max := A[i]

  return max

Na CRCW (z $n^2$ PE) algoritem zahteva konstanten čas ($O(1)$).

Kaj pa CREW? Ta ne dopušča sočasnega vpisovanja iz večih procesorjev. Tega CREW ne zmore v enem koraku, ker ne sme izpisovati več podatkov v isto lokacijo. CREW lahko to izvede v času, ki je kvečjemu $O(\log{n})$ (z zapisovanjem z drevesi).

Pri prehodu iz CRCW na CREW pridobimo faktor $\log{n}$.

Odnos CREW-EREW (patološki problem $P_2$)

Ali obstaja problem $P_2$, ki ga z enakim št. procesorjev CREW reši asimptotično hitreje kot katerikoli EREW?

Da. $P_2$ je problem pripadnost množici (ang. membership problem): Ali je dani $x$ element dane množice $\{x_1, \ldots, x_n\}$? Sprašujemo, ali je $x$ v množici.

Trditev: CREW PRAM (z $n$ PE) lahko reši $P_2$ v času $O(1)$.

Dokaz: Algoritem intuitivno dokažemo:

Vsakemu elementu $x_i$ je dodeljen svoj PE (označimo ga s $p_i$). Dana je spremenljivka $rez$, ki bo na koncu 1, če bomo za $x$ ugotovili, da pripada množici, oz. 0 sicer. Na začetku (ob inicializaciji) nastavimo $rez$ na 0. Vsak $p_i$ istočasno prebere $x$ (hkrati to lahko naredijo v CREW v času $O(1)$) in to primerja s svojim $x_i$. Če je rezultat primerjanja pozitiven ($x=x_i$), potem PE vpiše v rezultat rez = true (kvečjemu eden bo vpisal true v rez—to je dobro, ker imamo CREW).

Kaj pa EREW? Ne, v konstantnem času tega ne zmore. Potrebuje vsaj $\Omega(\log{n})$ korakov.

Težava: Kako naj si PE priskrbijo vsak svojo kopijo števila $x$ v času $O(1)$?

Kaj pa če bi bilo $n$ kopij že na voljo? Bi šlo, toda vstavljanje $n$ kopij zahteva $\lceil\log{n}\rceil$ korakov.

Izrek o simulaciji

Našli smo $P_1$ in $P_2$, ki ločita posamične modele PRAM za faktor vsaj $\Omega(\log{n})$. Temu faktorju rečemo ločevalni faktor (ang. separation factor).

Če gremo iz CRCW na bolj realen model CREW, plačamo ceno vsaj $\Omega(\log{n})$. Isto se zgodi pri prehodu iz CREW na EREW.

Vprašanje: Ali obstajajo $P'_1 in P'_2$, ki povzročita (imata) še večji ločevalni faktor? Torej, ali se modeli CRCW, CREW, EREW razlikujejo za poljubno velik razločevalni faktor? Ne. Razločevalni faktor je navzgor omejen, kar je dobro. Zgornja meja je kvečjemu $O(\log{n})$.

Modeli se med seboj razlikujejo natanko za $\log{n}$.

Izrek: Vsak vzporedni algoritem potrebuje na modelu CRCW PRAM (s $p$ procesorji) kvečjemu $O(\log{p})$-krat manj časa kot ga rabi na modelu EREW PRAM (s $p$ procesorji).

Praktična posledica tega izreka je:

Sestavi algoritem za CRCW PRAM (ker je lažje).
Analiziraj ga in določi $T_{par}$.
Ta algoritem bi tekel na EREW PRAM v času, ki je $T_{par} \cdot O(\log{p})$.

Ponovitev

Date: 2014-05-05

Vprašanje: Koliko je en model močnejši od drugega?

Če tisto kar zmore rešiti najhitrejši med njimi (CRCW) v času $t$, bo najpočasnejši (EREW) porabil $log(p) \cdot t$ časa oz. odvisen od logaritma števila procesorjev v najslabšem primeru. To je dobro, saj nas zanimajo poli-logaritmični algoritmi in zaradi tega ostajamo v istem prostoru.

Tehnika: Vzporedno urejanje (urejanje na PRAM)

Zaporedni alg.:

uredi $n$ števil
uporabljamo tudi operacijo primerjanja (se da tudi brez, npr. Bucket sort pri APS)
$T_{seq}(n) = \Theta(n \log n)$ (npr. Heap sort)
$\Theta(n \log n)$ je tudi spodnja meja za urejanje (dokaz pri APS)

Vzporedni alg.:

v kolikšnem času in s koliko PE se da urediti $n$ števil na vzporeden način?

Naj bo $S$ algoritem za vzp. urejanje na PRAM s $p$ procesorji.

Vemo že $C_p(n) = \Omega(T_{seq}(n))$ (cena alg. $S$ je navzdol omejena s časom najhitrejšega zap. alg. za urejanje). Iz definicije sledi:

\[ p \cdot T_{par}(p,n) = \Omega(n \log n) \]

Vzemimo $p = n$ in po deljenju z $n$: $T_{par}(n,n) = \Omega(\log n)$. Torej čas za vzp. urejanje $n$ števil je vsaj $\Theta(\log n)$. Ne vemo, če se da doseči, torej morda tudi asimptotično več.

Ali se da z $n$ PE urediti $n$ števil prav v času $\Theta(\log n)$? (Šteje le konstruktiven dokaz—t.j. konstrukcija takšnega algoritma.) Da. To je pokazal Richard Cole, 1986.

Primer: Cole-ov algoritem (skica)

Temelji na urejanju z zlivanjem (merge sort). Če je $n$ števil bo potrebnih $\log n$ vzp. zlivanj.

Primer: $\{1, 2, ..., 8\}$

Primer Cole-ov algoritem

Ker je vseh vzp. zlivanj $\log n$, bo čas $T_{par}(n,n) = \Theta(\log n)$ dosežen le, če bo vsako posamično vzp. zlivanje zahtevalo samo $O(1)$ časa. Kako? Izkoriščamo informacije od prejšnjih zlivanj (ranks, compare-exchange operations in hypercube).

Izrek (R. Cole): Zaporedje $n$ števil lahko uredimo na PRAM CREW z $\Theta(n)$ PE v času $T_{par}(n,n) = \Theta(\log n)$. (Na EREW PRAM pa $O(\log^2 n)$.)

Vendarle je v končni fazi/praksi vseeno smiselno pogledati čas izvajanja oz. konstante.

Osnovni razred vzp. zahtevnosti (pomen modela PRAM)

Def.: Računski problem je v razredu NC, če je rešljiv na modelu PRAM z $O(n^k)$ procesorjev v času $O(\log^c n)$.

Opomba: NC se po domače imenuje Nick’s Class (po Nick Pippenger).

Zakaj takšen pogoj? Če je v zap. svetu “hiter” algoritem le tisti, ki je polinomski, je v vzp. svetu “hiter” algoritem le tisti, ki je polilogaritmičen.

Teoretično lahko konstruiramo vzp. algoritme, ki uporabljajo eksponentno mnogo procesorjev (včasih je tak algoritem zelo hiter). Toda taki vzp. algoritmi v praksi tečejo bolj počasi kot trdi teorija! Vir težav je 3-dimenzionalen prostor realnosti. Četudi bi zgradili vzp. računalnik z eksponentnim številom PE in te zgnetli na najmanjši možni prostor, se povezave in PEji ne morejo zgnesti na ta prostor ne da bi se vsaj ena povezava bistveno podaljšala.

NC = razred rač. problemov, ki so v praksi hitro rešljivi na vzporeden način.

Veliko vprašanje vzp. računanja: $P \stackrel{?}{=} NC$ (ali nam realna vzporednost bistveno kaj prinese)?

Povzetek

PRAM je (videti) nerealističen model (ker ima neomejen skupni pomnilnik, takojšen dostop do vsake celice, zanemarja čas komunikacije) (danes že poznamo vrsto bolj realističnih)
kljub temu pa obstoj PRAM algoritma za dani problem:
1. pove nekaj o inherentnih lastnostih problema (npr. o njegovi dovzetnosti za paralelizacijo) (npr. polinomski GCD se upira paralelizaciji in je NC-poln)
2. lahko vodi do bolj praktičnega/realnega algoritma (za vzp. računalnik z dano arhitekturo, ki običajno namreč ni polni graf)
če nam ne uspe sestaviti za PRAM, ki je enostaven, je še manjša verjetnost da nam bo na kompleksnejšem modelu

Splošno mnenje: Le PRAM algoritmi, ki imajo optimalno ceno (tj. $C_p(n) = \Theta(T_{seq}(n))$) bodo v praksi morda relevantni. Med temi pa nas najbolj zanimajo tisti, ki imajo minimalen $T_{par}$.

Large networks grow smaller: How to choose the right simplification method?

2015-01-05T00:00:00+01:00

Conference proceeding

N. Blagus, L. Šubelj, G. Weiss, and M. Bajec, “Large networks grow smaller: How to choose the right simplification method?,” in Proc. of NetSci’14, 2014.

Abstract

Network simplification proved as an effective tool for reducing large real-world networks and at the same time providing for sufficient fit of original network. However, even though a number of analyses have been performed observing the changes of networks under the simplification, broad understanding of the whole process remains only partial. The questions such as ‘How to compare original (i.e., complete) and simplified (i.e., incomplete) network?’, ‘What factors impact the effectiveness of simplification process?’, ‘What size of simplified network provides for the best fit of original network?’, ‘What simplification method to use?’ are far from solved in the literature. In our study, we analyze over 30 real-world networks of different size and origin (e.g., social, information, technological). We reduce networks with several simplification methods (e.g., random node and link selection, breadth-first sampling, merging based on balance-propagation) and observe the changes of several fundamental properties (e.g., degree distribution, clustering coefficient, degree mixing, and density) under simplification. We show that the reduction on about 10% of original network provides for adequate preservation of important properties. The best performing methods prove to be random node selection based on degree and breadth-first sampling. The results also show the size of simplified network influence the effectiveness of simplification method, while the size and type of original network do not. Besides basic properties, we explore also the changes of network structure under simplification. Particularly, we focus on different groups of nodes, commonly observed in real-world networks (e.g., communities, modules and mixtures of the two). In this case, the changes in the simplification effectiveness occurs among different types of networks. For example, simplified social networks exhibit even stronger community structure than original networks, while in simplified information networks the number of mixtures increases. However, in general, the proportion of nodes explained by the group structure enlarge in sampled networks, moreover the goodness of the preservation of node group structure does not depend on the choice of the simplification method. To summarize, the main advantage of our analysis is large number of networks considered. Therefore we provide for reliable results concerning the effectiveness of the simplification process and support a better understanding of the changes of networks under the simplification process. In our future work we intend to create a framework for adaptive simplification of real-world networks, which would suggest the best simplification method for a given network based on its properties and the further use of the simplified network.

What coins the bitcoin? Exploratory analysis of bitcoin market value by network group discovery

2015-01-05T00:00:00+01:00

Conference proceeding

L. Šubelj, G. Weiss, N. Blagus, and M. Bajec, “What coins the bitcoin? Exploratory analysis of bitcoin market value by network group discovery,” in Proc. of NetSci’14, 2014, vol. 105, no. 2008.

Abstract

Bitcoin is a peer-to-peer digital currencies that gained tremendous popularity in recent years. Daily market trade currently exceeds 100000 BTC (over 50 million USD) with 70000 processed transactions. It relies on a network of computers that solve complex mathematical problems as a part of a process that verifies and permanently records every transaction made. Unlike traditional currencies, no central authority governs the supply or has control over bitcoins. The bitcoin market value thus depends solely on people’s confidence or trust in it, similar to other real-world commodities and assets. Consequently, little is known about the trading behavior on exchange markets and their periodic or mutual dynamics. We thus here report the results of a preliminary exploratory analysis of bitcoin market value from a popular exchange market BitStamp. We collect the data for a period of five days in January 2014 at a rate of about one minute and construct different network representation of the time series. In particular, we consider proximity networks including cycle and correlation networks, transition networks and visibility graphs. Cycle network has a grid-like structure with uniform degree distribution and high clustering. On the contrary, the transition network reveals very sparse topology with bell-shaped degree distribution and no pronounced degree correlations. Correlation network and visibility graph are scale-free and small-world with assortative mixing by degree, otherwise a characteristic of social networks. Next, we apply a node group discovery approach to constructed networks in order to gain an insight into the their structure, and to reveal possible periodic patterns and other dynamics of the time series. In fact, complex networks methods provide knowledge that is complementary to that of the standard approaches of time series analysis. We adopt group detection framework that can detect densely linked groups of nodes known as communities, groups of structurally equivalent nodes denoted modules, and different mixtures of these, with core/periphery and hub & spokes structures as special cases. According to the above, cycle network is too dense to reveal any clear structure. On the other hand, most significant groups in transition network are communities with high modularity that represent the quantiles occupied by the bitcoin market value for some period in time (6 hours). The latter could be used for concept drift detection, which is a difficult problem in stream mining. Groups in correlation network are core/periphery-like structures with negative modularity, where dense core correspond to periods in time with consistent bitcoin behavior and periphery are notable shifts in the value that appear rather periodically. Whether this could be adopted for market price prediction remains unclear. Finally, groups in the visibility graph show similar characteristics with periphery corresponding to local extremes in the value that dominate a certain period of time. The above network representations can also model multidimensional times series, which enables the analysis of bitcoin market value and trade from several exchange markets simultaneously. Since the value can differ substantively across the markets, predicting the future fluctuations at one market from the dynamics of another could be of considerable practical value.

[UI-part1] Ensemble methods for data analytics

2014-03-26T00:00:00+01:00

Course: https://ucilnica.fri.uni-lj.si/course/view.php?id=81
Lecturer: prof. Marko Robnik-Šikonja
Language: English
Date: 2014-02-26

Lecturers (3 parts):

prof. Marko Robnik Šikonja - Ensemble methods for data analytics
akad. prof. dr. Ivan Bratko - Learning in logic, ILP
izr. prof. dr. Zoran Bosnić - Incremental learning from data streams

Each part lasts 5 weeks:

2 weeks of lectures (~4 hours)
2 weeks of lab. exercises (~4 hours)
1 week for completing and delivering the seminar work
individual consultations (~25 hours of individual work)

Each part is to be completed with a grade at least 50%. The final grade is the average grade of all parts.

Course overview (part 1):

Ensemble methods are one of the most successful and general data mining techniques. The block presents relevant theory and practical approaches necessary to tailor ensemble methods to specific new tasks. Each student solves an individual assignment focused on problems from her/his research agenda.

Ensemble methods for data analytics

Introduction

R open source statistical system
Some exercises published (exercise.txt) for getting used to (source code for random forest is provided)
This years assignment (assignment.txt)
Instructions for project

Read at least all the abstracts from the Collection of papers on Spletna ucilnica. Also contains two highly recommended books (Hastie, Tibshirani, Friedman: “The elements of statistical learning”; Gareth et al.: “An Introduction to Statistical Learning with Applications in R”; another good one is R. E. Shepire, Y. Freund: “Boosting: Foundation of Algorithms”, 2012).

Grading (you need to get at least 50%):

project (ensemble learning in connection with your research agenda) [60%] - next week individually
presentation of project [20%] - 26.3.2014, 10 min presentation, submit report till 26.3.2014 8:00, 3-4 pages (problem description, data, related work, ensemble methods used)
smaller programming assignment [20%]

General scheme for ensemble learning (EL)

Pseudo-code:

construct $T$ nonidentical learners
combine their predictions (eg. by simple voting)

The learners should be nonidentical in the sense of predictions and at least weak learners ($\epsilon > 0$ better than noise).

Approaches covering specific variations (left are options that can be varyied and right algorithms doing that):

parameters: stacking
data instances: bagging, boosting
data attributes: random forests
learning algorithms: stacking

\[ D = \{(x_1, y_1), (x_2, y_2), ..., (x_n, y_n)\} \]

In real world problems the true generating model is unknown ($y = f(x) + \epsilon$, $\text{E}[\epsilon] = 0$).

With a learning algorithm we than train a model $g(x|D)$ to approximate $f(x)$. One way to calculate the error is mean square error:

\[ MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - g(x_i|D))^2 \]

If we try to compute the expected error:

\[ \text{E}[MSE] = \frac{1}{n} \sum_{i=1}^{n} \text{E}[(y_i - g_i)^2], g_i = g(x_i|D) \]

Bias variance decomposition can be used to decompose this expectation:

$\text{E}[(y_i - g_i)^2]$
$= \text{E}[(y_i - f_i + f_i - g_i)^2]$
$= \text{E}[(y_i - f_i)^2] + \text{E}[(f_i - g_i)^2] + 2 * \text{E}[(f_i - g_i)*(y_i - f_i)]$
$= \text{E}[\epsilon^2] + \text{E}[(f_i - g_i)^2] + 2 * (\text{E}[f_i*y_i] - \text{E}[f_i^2] - \text{E}[g_i*y_i] + \text{E}[g_i*f_i])$
$.. \text{E}[\epsilon^2] = 0$
$.. \text{E}[(f_i - g_i)^2] = \text{E}[f_i^2]$
$.. \text{E}[f_i*y_i] = \text{E}[g_i*(f_i + \epsilon)] = \text{E}[g_i*f_i + g_i*\epsilon] = \text{E}[g_i*f_i] + 0$
$= \text{E}[\epsilon^2] + \text{E}[(f_i - g_i)^2]$
$.. \text{E}[(f_i - g_i)^2] = \text{E}[(f_i - \text{E}[g_i])^2] + \text{E}[(\text{E}[g_i] - g_i)^2] + 0$
$.. \text{E}[a*X] = a*\text{E}[X]$

$\text{E}[(y_i - g_i)^2]$
$= \text{E}[\epsilon^2] + \text{E}[(f_i - \text{E}[g_i])^2] + \text{E}[(\text{E}[g_i] - g_i)^2]$
$= \text{Var}[\epsilon] + (bias)^2 + \text{Var}[g_i]$

To improve our model we can therefore improve the bias or variance of our model (variance $\epsilon$ in original data can not be improved). With ensemble learning we mostly try to reduce the variance. Different models have different biases (eg. decision trees work on rectangles in problem space). By combining different models you are also solving the bias problem, because averaged decision models are not restricted to biases anymore.

Error due to Bias: The error due to bias is taken as the difference between the expected (or average) prediction of our model and the correct value which we are trying to predict. Of course you only have one model so talking about expected or average prediction values might seem a little strange. However, imagine you could repeat the whole model building process more than once: each time you gather new data and run a new analysis creating a new model. Due to randomness in the underlying data sets, the resulting models will have a range of predictions. Bias measures how far off in general these models’ predictions are from the correct value.

Error due to Variance: The error due to variance is taken as the variability of a model prediction for a given data point. Again, imagine you can repeat the entire model building process multiple times. The variance is how much the predictions for a given point vary between different realizations of the model.

Scott Fortmann-Roe

Kinds of ensemble models

Sequential ensembles – eg. boosting (more control over the learning process, internally measure the error and focus on problematic instances; danger is over-fitting):

build a model
estimate error
reweigh the data
(repeat)

Parallel ensembles – eg. bagging, random forest (no estimation of error in between the process, parallelization; danger may be that you don’t get all special cases):

build lots of independent models
combine

Parallel ensembles

All are learned in parallel and their results are combined by voting.

Algorithm: Bagging (Leo Breiman, 1996)

Input:

dataset $D = \{(x_1, y_1), ..., (x_n, y_n)\}$
base learning algorithm $L$
number of base models $T$

Pseudo-code:

for(t in 1..T)
  h_t = L(D, D_bs)  // D_bs is bootstrap sample
end

(bootstrap sampling from $n$ items = select $n$ items with replacement)

Output:

$H(x) = \operatorname*{arg\,max}_y \sum_{t=1}^{T} I(h_t(x) = y)$
where $y$ is element from a set of classes $Y$, and: \[ I(statement) = \begin{cases} 1 & \text{if } statement = true \\ 0 & \text{else} \end{cases} \]

Usually the learning algorithm is a tree model learner, because they seem to be the most unstable (can produce different trees), simple, and fast learner. Final classification is the one selected most of the time by voting.

Why it works? Estimate error of whole ensemble.

$Y = \{-1, 1\}$
$H(x) = sign( \sum_{i=1}^{T} h_i(x) )$

$P(h_i(x) \neq f(x)) = \epsilon$
$= \sum_{k=0}^{\frac{T}{2}} \binom{T}{K}*(1-\epsilon)^k*\epsilon^{T-k} \leq e^{-\frac{1}{2}*T*(2*\epsilon-1)^2}$

Hoeffling inequality states:

$x_1, x_2, ..., x_n$
$\overline{X} = \frac{1}{n} * (x_1 + ... + x_n)$
$P(x_i \in [a_i, b_i]) = 1$

$P(\overline{X} - E[\overline{X}] \geq t) \leq e^{(-2*n^2*t^2) / \sum_{i=1}^{n} (b_i-a_i)^2}$

One sample: $(1-\frac{1}{e})*n$ different instances on average, using 63,2% of instances per model, ~37% of instances are not used in one learner (out-of-bag samples). Out-of-bag samples can be used for estimating the performance/error or controlling the learning process.

Algorithm: Random forests

People tried to increase the instability of learners to produce different models and random forests were invented. Algorithm is pretty similar to bagging, but the learning algorithm is the random tree learner L, and each execution of it is also given a different subset of attributes.

randomTree(instance):
  if num. of instances <= threshold:
    return leafNode
  select best attribute A from the random subset of size k
  split data according to A into left and right subset
  leftB = randomTree(leftSubset)
  rightB = randomTree(rightSubset)
  return tree

Threshold is typically $1$, so complete and random trees are built. For $k = \log_2{(\text{num. of attrs})}$ or $k = \sqrt{(\text{num. of attrs})}$.

We get many different trees, each one produces a vote and we sum together the votes. But they are mostly incomprehensible.

Additional information:

importance of attributes (for each tree we know used attributes and their oob set (out-of-bag samples), we put oob instances down the tree, we check what effect it has if we change or ignore an attribute, can also find conditional dependencies between attributes)

Margin is the distance between the classification condition and nearest instance, we want to maximize it (SVM does this explicitly). For random trees it is better to estimate the margin than classification errors.

Margin of random forest:

$mr(x, y)$
$= P(H(x)=y) - \max_{j \in C, j \neq y} P(H(x)=j)$
$= (\text{probability of correct class}) - (\text{probability of maximal incorrect class})$

Next time

bring laptops
1 hour trying to solve exercises
in the mean time individual consultations
try to figure out how ensembles can be used in your problem

Review

Date: 2014-03-05

Random forest:

different configurations
$n$ instances with replacement
not selected are out-of-bag instances (~37%)
additional randomness is introduced by selecting a subset of $log_2 a$ or $\sqrt{a}$ random sets in each node
almost no parameters, just num. of trees and num. of attributes ($a$)

Feature evaluation

Feature evaluation and estimation of generalization performance can be performed by randomly scrambling attributes/features:

keep distribution
if distribution performance is important than the classification performance (eg. accuracy, or better estimate the margins between classes) drops a lot, else the attribute in unimportant
for each feature we get a score that is an importance of this feature
we can compare it ReliefF

With random forest

distance measure: classify all trees, we count how many times two instances came into the same leaf, count co-occurrences in leaves
find outliers: compute distance to all other instances and the ones with largest distance is an outlier; we can represent is as a distance/similarity/proximity matrix between all pairs of instances, find a few largest distances (~0.5%, problem dependent), convert to distance values with $\sqrt{1 - \frac{\text{proximity}}{\text{num. of trees}}}$
multidimensional scaling: lower-dimensional projection of the data, identify clusters visually
proximity matrix enables you to do clustering, gives you intrinsic similarity, can expose class fragmentation problem (eg. a disease can be caused by many factors, people with a disease can have different reasons), you can also see if clustering matches your classes

Density estimation

Probability density visualization in 1D tells you where your instances would be (eg. Gaussian). Higher dimensions can be visualized with heat maps or histograms. Gaussian representation in more than 5D dysfunctional.

With random trees

Random trees with random splits until you get a leaf (full trees with single instance in the leaf). Reasonably good statistics can be estimated based on average depth of instances in leaves. For each instance you approximately know in what dense region it appears.

Sequential ensembles

First a classifier $C_1$, estimate error $e_1$, next classifier $C_2$ corrects the problems of previous classifiers, new error $e_2$ is estimated, and so on… On the end their results are combined with weighted voting. Such approach is also capable of reducing the bias. Family of best is called boosting.

Algorithm: AdaBoost

Input:

dataset $D = \{(x_1, y_1), (x_2, y_2), ..., (x_n, y_n)\}$
base algorithm $L$
number of training rounds $T$

Pseudo-code:

D_1(i) = 1 / n  // weight of instances
for i = 1 to T
  h_t = L(D, D_t)  // either directly using weights or sampling with D_f
  e_t = Pr(D_i; h_t != y_i)  // probability estimate hypothesis condition based on sample D_i
  a_t = 1/2 * ln((1 - e_t) / e_t)  // negative weight if bellow 1/2
  Z_t = Sum_{i=1}^{n} D_t(i)
  D_{t+1}(i) = D_t(i) / Z_t * e^(-a_t*y_i*h_t(x_i))
end

Output:

$H(x) = sign \sum_{t=1}^{T} a_t * h_t(x)$

Also:

for $a$ - if the error is a little bit above $\frac{1}{2}$, we can say we learned a little bit
where $Z_t = \sum_{i=1}^{n} D_t(i)$ is the normalization factor assuring a probability distribution of $D_t(i)$
also $D_{t+1}(i)$ equals: \[ D_{t+1}(i) = \frac{D_t(i)}{Z_t} * \begin{cases} e^{-a_t} & \text{if } h_t(x_i) = y_i \\ e^{a_t} & \text{if } h_t(x_i \neq y_i) \end{cases} \]

Small decision trees and decision stamps work fine and are pretty fast with AdaBoost. Noisy data results in AdaBoost over-fitting it (because of increased weights). Prevent this by limiting the maximal weight (eg. MadaBoost).

Explanation is harder than random forests. A general explanation method for predictions (E. Štrumbelj). Make a small modification in the attribute and you estimate its influence on the prediction. This way you can explain what influence each attribute has. From the perspective of game theory it is possible to show that the explanation is correct. Perturbation method with clever sampling.

Stacking

Basic models give you predictions for each instance and you add them as additional features (second level learner). In addition to basic attributes you also get additional features as predictions of some models.

Input:

dataset $D = \{(x_1, y_1), (x_2, y_2), ..., (x_n, y_n)\}$
first level learners $L_1, L_2, ..., L_t$ (any strong learners, eg. random forest, boosting, neural networks)
second level algorithm $L$ (frequently generalized linear model (GLM))

Pseudo-code:

for i = 1 to T {
  H_t = L_t(D)
}
D' = {}
for i = 1 to n {
  generate new dataset
  for t = 1 to T {
    Z_{i,t} = h_t(X_i)
  }
  D' = D' union {((Z_{i,1}, Z_{i,2}, ..., Z_{i,T}), y_i)}
}
h' = L(D')

Output:

$H(x) = h'(h_1(x), h_2(x), ..., h_T(x))$

Also:

we try to figure out where each classifier was wrong and improve that
can be seen as a generalization of bagging
if base learners are capable of outputting the probability, also probabilities of each class can be for each one ($p_1, p_2, ..., p_c$, $t * c + 1$ columns)

Assignment

$bias = E[(f_i - h_i)^2]$
$var = E[(h_i - \overline{h_i})^2]$
randomly select/sample from 1/2 of dataset
plot bias-variance decomposition

Clustering ensembles

Date: 2014-03-12

Clustering is about getting a general overview of data.

How to generate a single clustering?
- k-means, k-medoids, hierarchical, RF…
How to combine several clusterings?

Base clustering example:

$\lambda^{(1)} = \{1,1,2,2,2,3,3\}$
3 classes ($1: x_1, x_2$; $2: x_3, x_4, x_5$; $3: x_6, x_7$) (exactly same as $\lambda^{(2)} = \{3,3,1,1,1,2,2\}$).

Form a similarity matrix $M$ (dim. $n*m$), where $M(i,j)$ is similarity score of instance $x_i$, $x_j$ (eg. 1 if in same cluster, 0 otherwise).

Similarity ensemble for clustering

Input:

dataset $D = \{x_1,...,x_n\}$
clusterer $L$

Pseudo-code:

for(q = 1 to r) {  // r different clusterings
  lambda^{(q)} = L^{(q)}(D)  // form a clustering with k^{(q)} clusters
  produce similarity matrix M^{(q)} based on lambda^{(q)}
}
M = 1/r * sum_{q=1}^r (M^{(q)})  // consensus similarity

Output:

ensemble clustering: $\lambda = L(M)$

Algorithm: AODE (Averaged One-Dependence estimator)

Recall that Naive Bayes assumes independence that a class attribute $y \in \{c_1,c_2,...c_k\}$ is dependent on individual attributes:

\[ P(y|x_1,x_2,...,x_a) = \frac{P(y,x_1,...x_a)}{P(x_1,...x_a)} = \frac{P(y) \prod_{i=1}^{a} P(x_i|y)}{P(x_1,,...x_a)} \]

Only nominator part is relevant: $h(x) = \max_y P(y) \prod_{i=1}^{a} P(x_i|y)$

Relaxation of independence assumption and add just one attribute into the equation: $h_j(x) = \max_y P(y,x_j) \prod_{i=1,i \ne j}^{a} P(x_i|y,x_j)$

One classifier for each attribute, each attribute can be conditionally dependent on each classifier and from this ones we can build an ensemble. Eg. 10 classes and 20 values in an attribute, we have to assess 200 combinations, but the fragmentation of the data will increase and we need lots of data.

Averaging over all relaxed classifiers we get AODE:

\[ h(x) = \frac{1}{a} \sum_{j=1}^{a} h_j(x) \]

Algorithm: MARS (Multivariate Adaptive Regression Splines)

It is a variant of step-wise regression and it even works well for large number of attributes.

two basic functions (learners): $(x-t)_+$, $(t-x)_+$ (linear function, $t$ given point, $+$ indicates only positive parts)

Algorithm MARS functions

\[ (x-t)_+ = \begin{cases} x-t & \text{if } x > t \\ 0 & \text{else} \end{cases} \]

\[ (t-x)_+ = \begin{cases} t-x & \text{if } x < t \\ 0 & \text{else} \end{cases} \]

We put two such functions for $t$ in every value of every attribute and than we want to combine them into an ensemble. We combine all by selecting only the useful ones. We multiply basic functions together.

Idea:

collection $C = \{(x_j-t)_+, (t-x_j)_+; t \in \{x_{1j}, x_{2j},...x_{nj}\}; j \in \{1,2,...a\}\}$
$f(x) = \beta_0 + \sum_{m=1}^{M} \beta_m h_m(x)$
$h_m(x) \in C$
effectively we multiply all collections $C \times C \times ...C^i$
$\beta_m$ – learn coefficients learn a generalized linear model (using least squares criterion)
candidates consist of pairs of functions for each attribute value
by multiplying more and more line segments we approximate the target function more and more
we’ll probably over-fit it

Pseudo-code (iteratively create an ensemble members):

start with a constant ($1$)
repeat until (satisfied)
select best candidate multiplied with existing members
add to model
prune the model (using regularization or heuristic based on complexity)

Const-sensitive learning ensembles

Misclassification cost is the cost when we make an error. Represented as values in a matrix between predicted and actual classifications. Asymmetric misclassification cost matrix introduces cost-sensitive learning.

One re-sampling approach is done in boosting-type of algorithms where we push instances of high cost to classifiers.

Algorithm: MetaCost (based on bagging)

Input:

training data $D$
cost matrix $C$, $c_{ij}$ is the cost of misclassifying instance of class $i$ as class $j$

Pseudo-code:

for(i=1 to T) {  // T number of basic models
  D_i = subsample of D with n instances (bootstrap)
  M_i = L(D_i)
  for(each x in D) {
    for(each class j) {
      P(j|x) = 1/T * sum_{i=1}^{T} (P(j|x,M_i))
    }
  }
  assign a label to x as: arg min_{i \in \{1,...c\}} sum_{j \in \{1,...c\}} P(j|x)*C_{ji} (we weight all probabilities by their cost)
}

Output:

$L(\{\text{a new dataset of }(x, \text{assigned class values})\})$

Unbalanced dataset is where instances of one class are more probable than the other (eg. 1: 1%; 0: 99%). One possible approach is to use a cost matrix. Learning such data can result in overfitting.

Two approaches:

oversampling
- technique SMOTE: compute some neighborhood an linearly in between the instances of different classes create new instances, put them based on previous clustering, density
undersampling (throw away some data)
- search for instances near the border, other can be thrown away (similar to support vectors)

Algorithm: EasyEnsemble

Ideas is to repeat the under-sampling enough times $T \sim \frac{|N|}{|P|}$.

Input:

dataset $D$
minority class examples $P \in D$ (positive, outliers, precious)
majority class instances $N \in D$ (negative)
$T$ – number of subsets from $N$

Pseudo-code:

for(i=1 to T) {
  find a random subset N_i \in N with |N_i| = |P|
  use N_i and P to train AdaBoost ensemble H_i(x) = sign \sum_{j=1}^{S_i} a_{ij} * h_{ij}(x)
}

Output:

$H(x) = \operatorname{sign} \sum_{i=1}^{T} H_i(x)$

[AA-part1] Parallel architectures

2014-03-17T00:00:00+01:00

Course: https://ucilnica.fri.uni-lj.si/course/view.php?id=89
Lecturer: prof. dr. Dušan Kodek
Language: English, Slovenian
Date: 2014-02-24

Lecturers (3 parts):

prof. dr. Dušan Kodek - Parallel architectures, SIMD, MIMD
prof. dr. Borut Robič - Parallel algorithms, PRAM
doc. dr. Tomaž Dobravec - CUDA architecture, OpenCL

The purpose of this course is to introduce students to the field of parallel computing which is in many areas becoming a basic tool for problem solving. The topics include architectures of parallel computers as well as algorithms that are needed for this type of computation and are closely related to a given architecture. The structure of the course will allow students to use theoretical knowledge for the practical design of parallel computer systems and parallel algorithms that can be used for complex problem solving. The latest parallel computers will be studied as examples and the advanced tools for solving a typical parallel problem will be given.

Course overview (part 1):

limitations of serial computation: IPC, Tomasulo algorithm, superscalar and VLIW approach
development of parallel architectures and technology: vector, SIMD and MIMD
interprocessor communication problems and interconnect networks
review of architectures used in the most powerful parallel computers to date

Exams depend on the lecturers, are more individual, and you contact them when you are ready. Basic structure:

homework (seminar or written exam at home)
oral exam

Kontaktna oseba za litraturo (dve mapi) je as. A. Božiček. Oglasit pri prof. D. Kodeku, da dobiš domačo nalogo, ki jo je potrebno rešiti pred ustnim izpitom.

Introduction

Reasons for parallel computation

Transistors/chip: 1971 ~ $10^4 / \text{chip}$ .. 2014 ~ $10^{10} / \text{chip}$; ~1000000x more

Clock frequency (speed/transistor): 1971 ~ $1 \text{MHz}$ .. 2014 ~ $5 \text{GHz}$; ~5000x more

Chip size/speed: ~200x more

Number of transistors per chip doubles every 18 months.

Moore’s law (1965, 1975 corrected)

Amdahl’s law represents a limit on speedup.

We really have no choice.

Flynn classification (M. J. Flynn, 1966)

+-----------+ <--(instructions, m_I)---- +--------+
| Processor |                            | Memory |
+-----------+ <--(data/operands, m_D)--> +--------+

SISD (single instruction single data): $m_I = 1$, $m_D = 1$
SIMD (single instruction multiple data): $m_I = 1$, $m_D = N > 1$
MISD doesn’t exist
MIMD (multiple instruction multiple data, multicomputers): $m_I = M \gg 1$ (thousands), $m_D = N \gg 1$

Types of parallelism:

Instruction parallelism (difficult because threads need to be identified before parallelization)
Data/operand parallelism

In list of Top500 supercomputers Tianhe-2 is the current top.

Enormous heat problem on each switch of transistor, each wire works as a resistor (heat energy = $C * U^2 / 2$). Voltage is decreasing from $5 \text{V}$, $3 \text{V}$, …, $0.75 \text{V}$, but silicon transistors do not work bellow this. Power and heat give an upper limit on how big a processor can be.

Limitations of parallelism

Amdahl’s law (G. M. Amdahl, 1967):

\[ S(N) = \frac{1}{f + (1 - f) / N} \]

$N$ .. number of processors
$S(N)$ .. speedup
$f$ .. sequential part
$1 - f$ .. parallel part

Eg.: $f = 0.1 \rightarrow S(N) \leq 10$; $f = 0.001 \rightarrow S(N) \leq 1000$

Parallelization is not universal, it is good for some problems, but not for all problems (eg. simplex algorithm is difficult to parallelize).

SISD computers

Where is the problem?

pipeline, $CPI \geq 1$ (clocks per instruction close to 1)
pipeline hazards:
1. data hazard (unavailable operands):
  - read after write (RAW) (flow dependence)
  - write after write (WAW) (antidependence)
  - write after read (WAR) (output dependence)
2. control hazard (branch instructions)
3. stuctural hazard (busy functional units)

Elimination of data hazards

FDIV F0, F5, F6  ; F0<-F5/F6
FADD F4, F0, F2
FSUB F8, F2, F1

Register F0: RAW hazard (~1960) is a true data hazard and nothing can be done, except dynamic execution (reordering of instructions).

FDIV F0, F5, F6
FADD F4, F0, F2
FST 0(R1), F4
FSUB F2, F3, F7
FMUL F4, F3, F2

WAW and WAR hazards represent naming dependencies that can be solved by renaming registers (F2$\rightarrow$FT1, F4$\rightarrow$FT2):

FDIV F0, F5, F6
FADD FT2, F0, F2
FST 0(R1), FT2
FSUB FT1, F3, F7
FMUL F4, F3, FT1

Who should do this renaming?

programmer or compiler (software): same programs use many more registers, program can not adapt to newer processors with more registers
hardware solutions (should solve in $1$ instruction per $0.145 \text{ns}$):
- scoreboarding (1963, CDC 6600)
- Tomasulo algorithm (1967, IBM 360/91)

Tomasulo algorithm

(Chapter 7 in Kodek book.)

Recall that all processors (except simple) consist of multiple functional units (*, %, +/-, store, load) (sequential and parallel) that can also be pipelined or not. Each functional unit has a reservation station (FIFO or different). Waiting instructions IR are placed in one of the reservation stations if they are available (otherwise it waits) with either real or virtual values (until real values are available instruction is not executed).

Each reservation station has 6 parameters:

$O_p$ - operation to perform, each unit may have more operations
$U_j$, $U_k$ - addresses of the reservation station that will produce results (for those that are waiting)
$V_j$, $V_k$ - real values of 2 input operands/results
$R_j$, $R_k$ - flags indicating when $V_j$ and $V_k$ are ready
$B$ - busy flag indicates reservation station (if all are busy there is a structural hazard and the instruction must wait)

Output from the functional unit is placed on common data bus (CDB) and goes back to all reservation stations that check if they need the resulting value (format, value, $U_j$, $U_k$). The result also must also be written to programmer accessible registers of the architecture.

Implicit renaming is done at the issue stage when instructions are written into reservation stations. This resolved WAW and WAR hazards.

Processing stalls if the number of reservation stations for a functional unit is not high enough (or in case of jump instructions), but this rarely happens. Newer Intel processors have 50-60 places in reservation stations. The results get out in a different order, but renaming writes correct values.

What happens if there is a branch (jump)?

In this version everything stops and waits for it to complete. Improvement of Tomasulo’s algorithm allows speculative execution of instructions.

Speculation consists of:

branch prediction
speculative execution using a reorder buffer
once the branch is resolved:
1. correct prediction: ok
2. wrong prediction: remove results of speculative execution from reorder buffer

Implementation is done by adding a reorder buffer (FIFO queue) to Tomasulo’s algorithm, working with it instead of directly with programmer accessible registers, and commit results in order instructions were given. But even with this we still have $CPI \geq 1$. Newer computers (starting with Pentium Pro) use superscalarity (multiple-issue) and are capable of going to $CPI \approx 0.5$.

Ponovitev

Date: 2014-03-03

Za redke probleme primerna velika paralelnost: napovedovanje vremena, kemijske reakcije, analiza dogajanja pri atomskih eksplozijah…

Pohitritev SISD:

Tomasulov algoritem
špekulativno izvrševanje ukazov z ROB (preureditveni izravnalnik)

Še vedno pa je: $CPI \geq 1$, $IPC = \frac{1}{CPI}$

Večizvršitveni računalniki

Želimo doseči $IPC > 1$ brez spreminjanja programov.

Poznamo pristopa:

superskalarni procesorji
VLIW/EPIC procesorji (dolgi ukazi)

Primer:

$n$-izstavitveni procesor
$x_i = f(x_j, x_n)$
medsebojna odvisnost operandov (RAW, WAR, WAW)

Potrebno število kontrolnih primerjav:

$1$. ukaz $0$ primerjav
$2$. ukaz $2$ primerjavi
$3$. ukaz $2*2$ primerjav
$n$. ukaz $(n-1)*2$ primerjav

Skupaj: $2 * \sum_{i=1}^{n-1} i = 2 * \frac{(n-1)*n}{2} = n^2 - n$ primerjav

Superskalarni procesorji: strojno ugotavljanja/odpravljanje medsebojnih odvisnosti:

procesor, leto, prevzem - izstavi - izvrši, funkcijskih enot
IBM RS/6000, 1990, 2-2-2, 2
Pentium Pro, 1995, 3-3-3, 5
DEC Alpha, 1998, 4-4-11, 6
Pentium 4, 2000, 3-3-4, 7
Core 7, 2009, 6-6-4, 15, 30 primerjalnikov
VLIW procesorji (very long instruction word) (EPIC je kratica pri Intel Itanium): dolgi ukazi s fiksnim številom običajnih ukazov, ki se lahko izvršujejo paralelno

Gre za programsko ugotavljanje/odpravljanje medsebojnih odvisnosti, kjer prevajalnik ugotovi kako razporediti (sprememba arhitekture zahteva recompileanje). Predvsem uspešno pri signalnih procesorjih, drugje nerodno.

Omejitve paralelizma na nivoju ukazov

Če je res, da lahko povečamo zmogljivost tako, da povečujemo število hkrati izstavljenih ukazov, se pojavi vprašanje koliko je sploh možno, če omejitev ne bi bilo?

Zamislimo si idealen superskalarni procesor:

neomejeno število registrov
vse napovedi skokov so pravilne
nobenih zgrešitev v predpomnilnikih

Kakšen je $IPC$? Na množici programov SPEC92:

na idealnem: od 17.9 do 150 (povprečje $\approx 80$)
na $n = 50$: od 80 do 45
3% napaka pri napovedi skokov: od 45 do 23
pri 256 registrih: od 23 do 16
resnični procesorji imajo IPC: od 2 do 4

Proizvajalci ne povečujejo $IPC$ (kar bi profesionalni uporabniki želeli), ampak dodajajo več jeder (cores). To je lažje, pri marketingu hitrost preprosto pomnožijo, proizvajalci zmagali bitko za $IPC$…

Paralelnost na nivoju niti

Nit (thread) je zaporedje ukazov in pripadajočih operandov, ki se lahko izvršuje neodvisno od ostalih niti.

Programer je tisti, ki mora identificirati niti in poskrbeti za njihovo sinhronizacijo. Npr. diskretna Furierjeva transformacija, množenje matrik, seštevanje…

Vektorski računalniki (SISD)

Spadajo pod SISD, ne SIMD. Čeprav je en ukaz, se operacije izvedejo zaporedno.

\[ a(i) = b(i) + c(i), i=1..N \]

Pri njih izkoriščamo podatkovno (operandno) paralelnost. Prednosti:

elementi vektorja so skoraj vedno medseboj neodvisni (ni operandnih nevarnosti)
en vektorski ukaz pomeni $N$ operacij –> manj ukazov (Flynnovo ozko grlo se odpravi)
veliko manj kontrolnih nevarnosti pri skokih (manj zank)
dostop do pomnilnika je regularen –> pomnilniško prepletanje je učinkovito

Ozko grlo prenosa podatkov med CPE in pomnilnikom se rešuje s širšim vodilom in s pomnilniškim prepletanjem ($m$ modulov (npr. 16, 32, 64, 128, 256)).

Najbolj znana serija Cray (Cray 1, Cray X-M2…), ki so bili najhitrejši računalniki do začetka ~90-ih let. Te serije so imele sledeče funkcijske enote za operacije v plavajoči vejici (fiksna vejica ni relevantna):

+/-, 6
*, 7
1/x, 17
load, 9 do 17
store, 9 do 17
gather/scatter, 6

Zgradba:

vsak program ima skalarni in vektorski del
vektorski registri ($N = 64, 128, 256$) (nariše kot škatla z dvojno črto zgoraj in spodaj)
skalarni registri (nariše kot škatla)
vektorska funkcijska enota (more znati po vrsti jemati iz vektorskega registra, na vsakem izvesti operacijo in pisati nazaj v vektorski register po vrsti) (narišemo kot valj s črtami)
vektorska load/store enota
VL (vector length) register (je vhodni podatek vsem fukcijskim enotam, dolžina 1 je identično skalarnemu, a počasnejša)

Primer DAxPy double-precision (64-bit); SAxPy single-precision (32-bit) z $N = 64$:

\[ \vec{y} = a * \vec{x} + \vec{y} \]

Program na skalarnem računalniku:

LD F0, a
ADDI R4, Rx, #64
LD F2, 0(Rx)      :loop
MULTD F2, F0, F2
LD F4, 0(Ry)
ADDD F4, F2, F4
SD 0(Ry), F4
ADDI Rx, Rx, #1
ADDI Ry, Ry, #1
SUB R5, R4, Rx
BNZ R5, loop

($2+9*64 = 578$ ukazov, $578$ urinih period)

Program na vektorskem računalniku:

LD F0, a
LV V1, Rx         ; V1<-x
MULSU V2, F0, V1  ; V2<-F0*V1
LV V3, Ry         ; V3<-y
ADDV V4, V3, V2   ; V4<-V3+V2
SV Ry, V4

($6$ ukazov, $1+12+7+6+12+63 = 101$ urinih period)

Chaining: Procesor lahko začne izračun takoj, ko je en element operacije že naložen. Ni potrebno čakati, da se vektorski register do konca napolni. Med tem, ko se prvi shranjuje se en kasnejši še bere, a kljub temu ponavadi ne pride do težav.

Strip mining: Če imamo preveč podatkov, resnični vektor predstavimo kot mnogokratnik vektorjev dolžine $64$.

Problem pogojno izvršljivih ukazov

Fortran koda:

    do 100 i=1,64
      if(A(i) ne 0) then A(i) = A(i) - B(i)
100 continue

Izvajanje na vektorskem računalniku:

LV V1, Ra        ; V1<-A
LV V2, Rb        ; V2<-B
LD F0, #0        ; F0<-0
SNESV F0, V1     ; VM<-1 if V1(i)!=0 else 0; scalar not equal vector
SUBV V1, V1, V2
CUM              ; VM<-1 v vsa mesta
SV Ra, V1        ; A<-V1

VM register (64-bitni mask), $N=64$, $\{N-1,N-2,...,0\}$

Problem pri redkih matrikah ($\vec{K}$, $\vec{M}$ indeksna vektorja). Fortran koda:

    do 100 i=1,64
      A(K(i)) = A(K(i)) + C(M(i))
100 continue

Izvajanje na vektorskem računalniku:

LV V1, Rk        ; V1<-K
LVI V2, (Ra+V1)  ; V2<-A(K), gather
LV V3, Rm
LVI V4, (Rc+V3)  ; V4<-C(M)
ADDV V5, V2, V4
SVI (Ra+V1), V5  ; pomn<-A(K), scatter

Redukcija vektorjev

Date: 2014-03-10

    dot = 0
    do 10 i=1,64
10  dot = dot + A(i) * B(i)

Problema se ne da vektorizirat, če ne uporabimo trika za redukcijo vektorjev:

    do 10 i=1,64
10  Y(i) = A(i) * B(i)  ; vektorski ukaz
    dot = Y(1)          ; skalarni ukaz
    do 20 i=2,64
20  dot = dot + Y(i)    ; vektorski ukaz

Parametri za učinkovitost vektorskega računalnika (za primerjavo):

$N_\inf$ - GFLOPS, teoretična zmogljivost, ki predpostavlja, da so vsi vektorji neskončno dolgi in vse funkcijske enote znajo delati z njimi
$N_{1/2}$ - dolžina vektorja pri kateri se doseže $N_\inf / 2$
$N_v$ - dolžina vektorja pri kateri postane vektorsko računanje hitrejše od skalarnega

Npr. družina Cray kjer ista funkcijska enota za skalarne in vektorske ukaze:

| $N_v = 2$, pri vektorskem ukazu je potrebno na začetku dolžino prenesti v funkcijsko enoto

Zakaj niso PC-ji vektorski?

| Dandanes sicer SSE, vektorski.
| Pri vektorskih se izkorišča operandno paralelnost.
| Zakaj imajo vsi PC-ji enoto za operacije v plavajoči vejici, čeprav samo 10% programov uporablja. Nekoč koprocesor (<486), cenovno tako ugodnejše.
| Razlog: cena CPE (registri so dragi), dostop do pomnilnika (širše prenosne poti, DRAM za skalarne dovolj)

GPU so postali popularni za GPGPU, a težko programirati, alternativa pa je Xeon Phi, ki ima 61-procesorjev in so programsko združljivi. Bomo videli kaj bo prevladalo.

Družina Cray:

| 1976  Cray-1
| ...
| 1996  Cray T90
| 2006  NEC
| MIMD  vektorski

Gordon Bell Prize (nagrada za računalnik, ki rešuje resničen problem, običajno manj kot polovica teoretične) - realne hitrosti:

| 1988  Cray Y-MP                    1 GFLOPS  zadnjič vektorski
| 1990  CM-2                        14 GFLOPS  zadnjič SIMD
| 2000  Grape-6                   1349 GFLOPS  MIMD
| 2010  Cray XP Jaguar            2330 TFLOPS  MIMD
| 2013  IBM Sequoia (BlueGene/Q)  14.4 PFLOPS  MIMD
| en core sodobnega PC teoretično 10

SIMD računalniki

Operandna/podatkovna paralelnost, ki jo je veliko lažje izkoriščati kot ukazno, saj programer že z definiranjem podatkovnih tipov omogoči izkoriščanje. Pri SIMD se ukazi v resnici izvedejo naenkrat, ne zaporedno kot pri vektorskih.

Vsi procesorji (imamo maskiranje) izvedejo isti ukaz na $n$ operandih naenkrat. Imamo gosteči računalnik, ki se ukvarja z vhodom in izhodom, saj nima smisla, da se SIMD specializiran za računanje ukvarja s tem.

Arhitektura SIMD računalnikov

PE_i - Procesorski element/procesor
PEM_i - Pomnilnik procesorske enote, ki vrača vrednosti nazaj v pomnilnik ali gosteči računalnik (za V/I)
Povezovalna mreža za komunikacijo med PE

Dva tipa:

distributed memory: povezovalna mreža za direktno komunikacijo med PE (posebni ukazi, čas dostopa krajši), NUMA - non-uniform memory access (čas glede na naslov ni enak)
shared memory: povezovalna mreža je povezana med PE_i in M_i ter se obnaša isto kot pomnilnik, ukazi za medprocesorsko komunikacijo niso potrebni

Večina jih je prvega tipa, saj je dostopanje do namenske povezovalne mreže hitrejše (podobno kot pomnilniška hierarhija).

Povezovalne mreže

Obstajajo problemi kjer je medprocesorske komunikacije malo (npr. računanje DFT), a na žalost niso vsi taki. Problem je kako narediti takšno komunikacijsko mrežo kjer lahko vsak komunicira z vsakim hitro.

\[ R_i -> R_{f(i)}, i=0,1,...,n-1 \]

Povezovalne mreže stopnja 1-4

Stopnja 1: vodilo/bus, problem le en lahko naenkrat komunicira z enim

Stopnja 2: pot ali cikel, vsak povezan s sosednjima, problem so konfliktni dostopi, kjer je potrebno dostopati

Stopnja 3: binarno drevo, problem pri komunikaciji med skrajnima listoma; izboljšava je debelo drevo, kjer imamo v višjih vejah/deblu po več povezav s čimer zmanjšamo verjetnost konfliktov

Stopnja 4: mreža/mesh, NEWS (north-east-west-south), 2D torus (avtomobilska guma, če še zunanje povežemo), vsako vozlišče povezano s 4 sosedi, pri komunikaciji med procesorji je linerana zveza skozi koliko vozlišč mora informacija potovati

Stopnja 4 je postala popularna, ker je povezana z naravo velikega števila problemov (npr. vreme, potovanje vode), kjer se prostor razdeli na posamezne dele in noben dogodek ne deluje na daljavo ne da bi šel čez vmesne dele. Obstajajo pa tudi problemi matematične narave, kjer ni teh omejitev.

Povezovalne mreže stopnja 5

Stopnja 5: 3D torus, kot več NEWS ravnin ena nad drugo, vsako vozlišče povezano s 6 sosedi

Povezovalne mreže stopnja n

Spremenljiva stopnja $n$:

hiperkocka ($n$-kocka): število vozlišč skozi katere more iti sporočilo je $log_2 n$, ni veliko takšnih (npr. Connection Machine 2)

To so bile statične mreže, obstajajo pa tudi dinamične mreže, kjer imamo stikala in se lahko povežejo po svoje. Pri Butterfly lahko sestavimo poljubno mrežo, če imamo stikalo, ki je sposobno potezati na oba načina.

Zgled: Illiac 4

Illiac 4 (1966-1972) izdelovali na Uni. Illinois, podjetje Westinghouse. Nekateri obtožujejo ta računalnik, da je upočasnil razvoj paralelnih računalnikov.

cilj bil doseči 1000 MFLOPS (32-bitnih)
4 kvadrante s po 64 PE/kvadrant
realizirali le 1 kvadrant
kontrolna enota brez pomnilnika, 64 PE s PEM, gostujoči PC B6500
problem feritni pomnilnik, stroški ogromni (porabili 4x več)

Arhitektura Illiac 4

Hitrosti operacij:

+/- 350ns
* 450ns
/ 2700ns
load/store 350/300ns

Posledično: < 3MFLOPS/PE x 64
teoretično = 180MFLOPS
dejansko = 15MFLOPS

Primeri kaj se zgodi, če ljudje nekaj spregledajo…

program čas rač. celoten čas
I4TRES 10800s 22100s
2D-TRANSONIC 1110s 2000s
SAR 28s 52s

Problem je bil prepočasen V/I (prenosi v in iz pomnilnika), prehitro računal.

Zgled: Connection Machine 2

Connection Machine 2 (1987), naredili več kot 50, podjetje TCM (Thinking Machines Corporation). Ideja je bila naredit računalnik, ki je samo “number cruncher”, a uporaben za reševanje različnih problemov.

$2^{16}$ 1-bitnih procesorjev
glavni pomnilnik imenovan Nexus
4 ali več gostečih (front-end) računalnikov, ki pripravlja stvari preden jih da v delu
$4$ kvadrantov, vsak s $2^{14} = 16384$ procesorji, povezani z Nexus
poseben V/I sistem z 1 do 8 “data vaults” (narejen iz diskov), da lahko podatke dovolj hitro prebere

2^4 = 16 1-bitnih procesorjev/plošči = 1 vozlišče
2^{16}/24 = 2^{12}, 12-hiperkocka
2 x 16 imamo 32-bitna enota za operacije v plavajoči vejici
2^11 32-bitnih operacij (na koncu uporabljalo za to, “number crunching”)
teoretična zmogljivost = 20GFLOPS

Vsi paralelni računalniki so za paralelne probleme, ne dobri za vse.

Prevlada MIMD

Po leti ~1990 so SIMD računalniki izginili, a zakaj?

Razvoj posebnih elementov za dano arhitekturo, saj je vsak SIMD je narejen drugače.
Razširljivost (scalability), kjer uporabnik želi velikost svojega računalnika prilagajati problemu, ki ga rešuje.
Razvoj tehnologije, saj ko so splošno namenski procesorji (off-the-shelf componenets) postali tako zmogljivi kot posebej narejeni in se ni več izplačalo delati SIMD.

Računalniku nič ne škodi, če deluje na MIMD način, le bolj splošen je. Tudi pri MIMD srečamo shared in distributed memory.

Problemi, ki se rešujejo običajno še vedno izkoriščajo operandno paralelnost, torej kot SIMD, a zmorejo tudi več.

Pregled

Date: 2014-03-17

SISD
Superskalarni
Tomasulo algoritem
Vektorski
SIMD

MIMD računalniki

Zakaj so izginili SIMD?

Tehnološki razvoj
- po letu ~1990 so “standardni” (off-the-shelf) procesorji postali dovolj zmogljivi.
Razširljivost SIMD je slaba
- povečevanje zmogljivosti z dodajanjem procesorjev
- povečanje zakasnitev (latence)
- povečevanje cene
- fizična velikost

Arhitektura MIMD računalnikov

Osnovni vrsti MIMD:

Skupen pomnilniški prostor
- tesno sklopljen sistem
- shared memory
- UMA (uniform memory access)
- vsak procesor $P_i$ ima svoj predpomnilnik $C_i$, ki so povezani v skupno povezovalno mrežo, le-ta pa na pomnilnik $M_i$
Več pomnilniških prostorov
- rahlo sklopljen sistem
- message passing
- distributed memory
- NUMA (non-uniform memory access)
- vsaj procesor $P_i$ ima svoj predpomnilnik $C_i$ in pomnilnik $M_i$, več pomnilniških prostorov, ki so šele potem povezani v povezovalno mrežo

Zgled: Tianhe-2

Lestvica TOP500 (teoretična zmogljivost):

2010 Tianhe-1A 4,7 PFLOPS 4,1 MW debelo drevo
2011 K computer 11,3 PFLOPS 12,6 MW “6D” torus (3D + dodatki)
2012 Titan-Cray XK7 27,1 PFLOPS 8,3 MW 3D torus
2013 Tianhe-2 54,9 PFLOPS 17,8 MW debelo drevo (+7 MW hlajenje)

Zgradba:

plošča (board):
- 2 modula:
  - CPM: 2x Ivy Bridge Xeon + 1x Xeon Phi)
  - APU (accelerated proc. unit): 5x Xeon Phi
  - mreža: 2x Gigabit LAN + TH-Express 2
- 2 vozlišči/ploščo: 64 GB DRAM (Ivy Bridge) + 24 GB DRAM (Xeon Phi) = 88 GB/ploščo
frame: 32 boards
cabinet (rack): 4 frames = 128 boards
computer: 125 cabinets (+ 13 za mrežo + 24 storage (diski))
vodno hlajenje
cena: ~500 M$

Heterogeni procesorji:

32000x Ivy Bridge Xeon E5-2692 2,2 GHz 12 jeder 211 GFLOPS/čip 115 W
48000x Xeon Phi 3151P 1,1 GHz 57 jeder 1003 GFLOPS/čip 270 W
Skupno jeder: $32000 * 12 + 48000 * 57 = 3120000 jeder$ (procesorjev)
Teoretična hitrost: $32000 * 0,211 + 48000 * 1,003 = 54896 PFLOPS$

Diski: 12,4 PB, H2FS hierarhija

Mreža: debelo drevo

Primerjava: Grid/cloud computing

Folding@home 2011 6 PFLOPS
Folding@home 2013 12 PFLOPS
SETI@home 2013 15,4 PFLOPS
BOINC 2013 9,2 PFLOPS
Bitcoin 2014 414,5 EFLOPS

Majhna količina/potreba medprocesorske komunikacije, a takšnih primerov je zelo malo.

Cluster (gruča) je skupek istih računalniko povezanih skupaj.

Primerjava: GPU computing

Računanje na grafičnih procesorjih (MIMD). Prvotno specializirani za operacije za grafiko, dandanes imamo standard CUDA za lažjo uporabo.

Intelov odgovor na to je koprocesor npr. Xeon Phi, ki je združljiv in uporablja skoraj enake ukaze kot prej. Zmoljivost se drastično poveča: 16 FLOPS/cycle/core = 1003 GLOPS/core, 2x večja poraba, a hitrost 5x večja.

Kaj bo prevladalo? Koprocesorji ali GPU? Bomo videli.

Energetska učinkovitost

Tianhe-2 ~ $3 GFLOPS/W$

Lestvica Green500:

nov. 2013 4,503 GFLOPS/W 28 kW
3,632 GFLOPS/W 52 kW
3,518 GFLOPS/W 78 kW

Kako narediti ExaFlops ($1000 PFLOPS$) računalnik?

Problem za segrevanje je število preklopov, ker pri zniževanju napetosti najprej hitrost pade, pod $0.7 V$ tranzistorji nehajo delati. Sodobni procesorji prilagajajo napetost/hitrost glede na obremenitev, temperaturo, in podobne faktorje.

V laboratorijih že imajo tranzistorje na drugačnih osnovah ($0.3 V$), a daleč od produkcije.

Podatkovno pretokovni računalniki (dataflow)

Obravnavali smo von Neumannove računalnike na katerih so od leta 1945 zasnovani vsi računalniki. Von Neumannov model je ukazno pretokovni (control flow) (strogo zaporedje: prevzem ukaza, izvršitev ukaza). Operand je točno določen z naslovom (oz. virtualnim naslovom).

Podatkovno pretokovni uporablja popolnoma drugačen pristop (ideja ~1950-ih, več pogovarjali v obdobju ~1960-1980). Problem lahko predstavimo kot graf operacij, npr:

\[ a = (b + 1) * (b - c) \]

Izvajanje dataflow računalnikov

operacijski paket
podatkovni žeton (token)

\[ \gamma: + (b) (1) \beta/1 \alpha: - (b) (c) \beta/2 \beta: * (\beta/1) (\beta/2) \delta/1=a \]

Podatkovni žetoni gredo v operacijske pakete na ustrezna mesta, njihov rezultati gredo naprej na ustrezna mesta v druge operacijske pakete.

Delovanje:

Data driven: operacija se izvrši, ko so prisotni vsi vhodni podatki
Demand driven: enako kot 1. + operacijski paket potrebuje rezultat

Operacije:

funkcija $f$, (krog, več vhodov, en izhod)
odločitev, (diamant, več vhodov, en izhod)
vrata $T$, spusti skozi, ko true, (krog s puščico, en vhod, en izhod)
preklop $T/F$, if stavek izbere med dvema vhodoma, (krog s puščico, dva vhoda, en izhod)
preklop $T/F$, if stavek izbere izhod, (krog s puščico, en vhod, dva izhoda)

Operacije dataflow računalnikov

Tipi:

statični podatkovno pretokovni rač. (1974 J. Damis):
- na loku grafa je lahko največ 1 žeton
- operacija se izvrši, če so prisotni vhodni operandi + prazen lok
dinamični podatkovno pretokovni rač. (1982 Manchester; 1983 Amind, MIT:
- na loku grafa je lahko več žetonov
- žetoni morajo biti oštevilčeni (obarvani)

Napake glede naslovov se ne morejo zgoditi, saj pojem naslova ne obstaja in ga po pomoti ni možno pokvariti (eg. stack overflow). V grafih nobene omejitve glede zaporednosti.

Kako naredimo takšen računalnik? (podobno je paketni mreži)

gosteči računalnik (zaradi lažje implementacije)
V/I stikalo nadzira gosteči računalnik
vsakemu podatku se glede na vrsto priredi podatkovni žeton
primerjalna enota vsebuje program oz. operacijske pakete, čaka na ustrezne žetone
operacijski paketi z vsemi žetoni nato potujejo v pomnilnik oz. čakalno vrsto
le-ti gredo nato v množico procesorjev iz katere pridejo žetoni rezultatov, ki spet po potrebi zaokrožijo

Arhitektura dataflow računalnikov

Problemi:

zmogljivost primerjalne enote, število primerjalnikov v malo večjem asociativnem predpomnilniku zelo naraste
paketna mreža ima enake probleme kot vse mreže, so lahko ozke mreže in procesorji lahko čakajo
velika “poraba” pomnilnika, vsaka spremenljivka ima lahko v nekem trenutku veliko število žetonov (npr. operacije z matrikami, kjer lahko iz ene vrednosti nastane več)

Takih računalnikov na trgu ni, razen za posebne namene. Kljub temu, da je ideja sicer zanimiva in nekaterih težav z njimi ni (stroge zaporednosti, segmentation fault), se niso obnesli, saj niso bolj zmogljivi.

Ostali:

analogni računalniki (z žicami, 1% natančnost), hibridni (lažje programirat)
nevronske mreže (pripravijo na PCju, lahko specializirana vezja), primerni za določene primere (npr. razpoznavanje obrazov)

Research tool nets-nodegroups

2015-07-08T00:00:00+02:00

Research tool nets-nodegroups implements the node group extraction framework for network analysis and introduces the group type parameter Tau for researching and exploring node group structures of various networks. The framework is capable of identifying and sequentially extracting significant node group structures, such as:

communities
modules
core/periphery
hubs & spokes
and many similar structures

Details of the algorithm and framework are published in:

L. Šubelj, N. Blagus, and M. Bajec, “Group extraction for real-world networks: The case of communities, modules, and hubs and spokes,” in Proc. of NetSci ’13, 2013, p. 152.
L. Šubelj, S. Žitnik, N. Blagus, and M. Bajec, “Node mixing and group structure of complex software networks,” Advs. Complex Syst., vol. 17, 2014.

The adopted network node group extraction framework extracts groups from a simple undirected graph sequentially. An optimization method (currently random-restart hill climbing) is used to maximize the group criterion W(S,T) and extract group S with the corresponding linking pattern T. After extraction edges between S and T are removed and the whole process repeated on the largest weakly-connected component until the group criterion W is larger than expected on a Erdös-Rényi random graph.

Open source project:

home: http://gw.tnode.com/nets-nodegroups/
github: http://github.com/gw0/nets-nodegroups/
technology: C++, SNAP library
citation:

Usage

First compile the tool to get the executable nodegroups (see below). To use it you need your file with graph edges and specify parameters you want to tweak:

-o: prefix for all file names (simplified usage) (default: graph)
-i: input file with graph edges (undirected edge per line) (default: graph.edgelist)
-l: input file (optional) with node labels (node ID, node label) (default: graph.labels)
-og: output file with ST-group assignments (default: graph.groups)
-os: output file with only ST-group extraction summary (default: graph.groupssum)
-n: number of restarts of the optimization algorithm (default: 2000)
-sm: maximal number of steps in each optimization run (default: 100000)
-sw: stop optimization if no W improvement in steps (default: 1000)
-ss: initial random-sample size of S ant T (0=random) (default: 1)
-fn: finish after extracting so many groups (turn off random graphs) (default: 0)
-fw: finish if W smaller than top percentile on random graphs (default: 1)
-rg: random graphs (Erdos-Renyi) to construct for estimating W (default: 500)
-rn: random graph restarts of the optimization algorithm (default: 10)
-rf: random graph re-estimation of W if relative difference smaller (default: inf)

Example to read from graph.edgelist and output to graph.groups and graph.groupsreport:

$ ./nodegroups -o:graph

Same example but extract only first 12 groups (ignoring estimated W on random graphs):

$ ./nodegroups -o:graph -fn:12

Input file graph.edgelist contains undirected graph edges (-i:):

Output file graph.groups contains extracted node groups S and linking patterns T (-og:):

# NId GroupS GroupT NLabel
0     0      0      foo
1     0      0      bar
2     0      0      foobar
3     1      -1     -
2     -1     1      -
...

Output file graph.groupsreport contains a summary of extracted node groups (-or:):

# Graphs: 12  Nodes: 115  Edges: 613
N   M   N_S M_S N_T M_T N_ST M_ST L_ST L_STc W        Tau    Mod_S  Mod_T  Type
115 613 9   36  9   36  9    36   72   25    823.0000 1.0000 0.1352 0.1352 COM
115 582 9   36  9   36  9    36   72   30    818.0000 1.0000 0.1164 0.1164 COM
115 550 10  40  10  40  10   40   80   30    810.0000 1.0000 0.1191 0.1191 COM
...

Description of columns:

N: number of nodes left in graph
M: number of edges left in graph
N_S: number of nodes in subgraph on group S
M_S: number of edges in subgraph on group S
N_T: number of nodes in subgraph on linking pattern T
M_T: number of edges in subgraph on linking pattern T
N_ST: number of nodes in subgraph on intersection of S and T
M_ST: number of edges in subgraph on intersection of S and T
L_ST: number of edges L(S,T) between groups S and T
L_STc: number of edges L(S,Tc) between groups S and complement of T
W: group critetion W(S,T)
Tau: group type parameter Tau(S,T)
Mod_S: modularity measure on group S
Mod_T: modularity measure on linking pattern T
Type: human name for group type parameter Tau:
- COM: community (S = T)
- MOD: module (S intersection with T = 0)
- HSD: hub&spokes module (module and |T| = 1)
- MIX: mixture (otherwise)
- CPX: core/periphery mixture (S subset of T or T subset of S)

Exit status codes:

-1: error, file not found or crash
0 : success, groups extracted
1 : no groups extracted

Build

You will need to compile the source code to get a working tool. This should work on any platform using any standard C++ compiler (eg. GCC, Visual Studio), but it has only been extensively tested in the following environment.

Debian (7.3, 8.0)
build-essential (~11.5)
g++ (~4:4.7.2-1)
SNAP (~2.1)

First you need to download source code of net-nodegroups:

$ git clone http://github.com/gw0/nets-nodegroups.git
$ cd ./nets-nodegroups

Download and compile SNAP library from console using GCC:

$ apt-get install build-essential g++
$ wget http://snap.stanford.edu/releases/Snap-2.1.zip
$ unzip Snap-2.1.zip
$ mv Snap-2.1 snap
$ cd ./snap
$ make all
$ cd ..

Compile net-nodegroups from console using GCC:

$ cd ./src
$ make all

Executable file is located in ./src/nodegroups.

Feedback

If you encounter any bugs or have feature requests, please file them in the issue tracker, or even develop it yourself and submit a pull request on GitHub.

License

This code is licensed under the GNU Affero General Public License 3.0+ (AGPL-3.0+). Note that it is mandatory to make all modifications and complete source code publicly available to any user.

Darky's ROM on Galaxy S i9000

2012-08-07T00:00:00+02:00

Darky’s ROM 9.2/10.2 is a custom ROM based on Android 2.2.1 and intended for phone Samsung Galaxy S (GT-I9000XXJPY). Here are brief instructions related to installing, hacking, and improving your mobile phone.

Flashing

Full backup from Linux

It is always recommended to do a full backup of the original firmware images before you start doing anything. To do this on a brand new Samsung Galaxy S with Android Recovery <3e> over USB without flashing:

if you plan to give images to someone else, do a factory reset
connect over USB in debugging mode and check if adb is working (part of Android SDK Tools)
download and extract root, backup, and clean scripts
root it with ./doroot.sh
make full backup images of all partitions with ./dobackup.sh
check if everything is fine and all images are present

Flashing Darky’s ROM 10.2 from Linux

Follow this instructions to flash a clean Darky’s ROM 10.2 on a Samsung Galaxy S from Linux:

if your ROM uses lagfixes, disable them
connect over USB
install heimdall tool (note to never use heimdall dump mode)
download and extract Darky’s ROM 10.2 Ressurection Edition
also extract PDA.tar.md5 inside archive (tar vxf PDA.tar.md5)
reboot into download mode (turn off and press volume up + home + power)
flash with heimdall:

$ heimdall flash --repartition --pit s1_odin_20100512.pit --factoryfs factoryfs.rfs --cache cache.rfs --dbdata dbdata.rfs --primary-boot boot.bin --secondary-boot Sbl.bin --param param.lfs --modem modem.bin --kernel zImage

wait for flashing and installation to finish
reboot once more

In Windows a similar tool for flashing the Samsung Galaxy S is called Odin (it needs files packed in a .tar, PDA).

Replace boot animation and disable boot sound

Darky’s ROM has a custom boot animation with sound that can be annoying. Instructions to replace the boot animation from Linux:

connect over USB in debugging mode and check if adb is working (part of Android SDK Tools)
download Nexus S boot animation (or another compatible animation)
replace boot animation:

$ adb shell mount -o remount,rw /dev/block/stl9 /system
$ adb shell mv /system/media/bootanimation.zip /system/media/bootanimation.zip.old
$ adb push bootanimation.zip /system/media/bootanimation.zip
$ adb shell chmod 644 /system/media/bootanimation.zip

disable boot sound:

$ adb shell mv /system/etc/PowerOn.wav /system/etc/PowerOn.wav.old
$ adb shell mv /system/darkysound/android_audio.mp3 /system/darkysound/android_audio.mp3.old

Clean Darky’s ROM

Unfortunately Darky’s ROM 9.2/10.2 contains a couple of useless or suspicious apps. Some of them can be uninstalled with the normal approach (under Settings/Apps menu):

Voodoo Control App
FasterFix

More permanent apps must be removed using ADB:

connect over USB in debugging mode and check if adb is working (part of Android SDK Tools)
remove apps:

$ ./adb shell su -c 'mount -o remount,rw /dev/block/stl9 /system'
$ ./adb shell mkdir /sdcard/device-files
$ ./adb shell su -c 'cp /system/app/Fileindex.apk /sdcard/device-files/ &amp;&amp; rm /system/app/Fileindex.apk'
$ ./adb shell su -c 'cp /system/app/PressReader.apk /sdcard/device-files/ &amp;&amp; rm /system/app/PressReader.apk'
$ ./adb shell su -c 'cp /system/app/Samsung.apk /sdcard/device-files/ &amp;&amp; rm /system/app/Samsung.apk'
$ ./adb shell su -c 'cp /system/app/Telegraaf.apk /sdcard/device-files/ &amp;&amp; rm /system/app/Telegraaf.apk'
$ ./adb shell su -c 'cp /system/app/txtr-android-client-bol-1.0.3-nobooks.apk /sdcard/device-files/ &amp;&amp; rm /system/app/txtr-android-client-bol-1.0.3-nobooks.apk'

Other

Unlock a screen locked device

If you were playing around too much with the pattern unlock screen on an Android phone, it will permanently lock. You then either need to unlock it with your Google account or using ADB (if you have USB debugging enabled):

connect over USB in debugging mode and check if adb is working (part of Android SDK Tools)
install sqlite3 tool on the phone or computer (in case it is not yet on your phone)
for Android versions with sqlite3 on the phone itself, you can do the following over ./adb shell on file /dbdata/databases/com.android.providers.settings/settings.db
for Android versions without sqlite3, an ugly solution is to download the problematic settings file ./adb pull /dbdata/databases/com.android.providers.settings/settings.db
open settings with ./sqlite settings.db
update security lock setting update secure set value=0 where name='lockscreen.lockedoutpermanently';
close settings .quit
in case you downloaded it, upload the settings file ./adb push settings.db /dbdata/databases/com.android.providers.settings/settings.db
to increase the number of failed login attempts before locking ./adb pull /data/system/device_policies.xml, modify the value in the XML file and upload it back

Installation alternatives for Debian 8

2015-05-15T00:00:00+02:00

Debian 8 and similar systems can be installed in many ways, but the alternative Network boot variant is the smallest and cleanest, as it installs directly from the internet. Depending on the selected method and architecture (amd64 is for 64-bit Intel/AMD) you need an appropriate installation image or special boot files.

Locally with network boot

From USB stick

insert an empty USB stick
download a Netboot ISO image

$ wget http://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-8.1.0-amd64-netinst.iso

write image to the USB stick (device /dev/sdX) as root

$ sudo dd if=debian-8.1.0-amd64-netinst.iso of=/dev/sdX bs=16M; sync

reboot from the USB stick (optionally add kernel parameter priority=low)

From CD/DVD

insert an empty CD or DVD
download a Netboot ISO image

$ wget http://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-8.1.0-amd64-netinst.iso

burn image (file debian-8.1.0-amd64-netinst.iso) onto the CD or DVD (using K3b or other CD/DVD burning utility)
reboot from the CD/DVD (optionally add kernel parameter priority=low)

From HDD using a special image

insert an empty target HDD
download a HD-media image

$ wget http://ftp.debian.org/debian/dists/jessie/main/installer-amd64/current/images/hd-media/boot.img.gz

write image it to the target HDD (device /dev/sdX) as root

$ gzip -cd boot.img.gz | sudo dd of=/dev/sdX bs=16M; sync

insert the HDD in the target machine
reboot from the HDD (optionally add kernel parameter priority=low)

From HDD using GRUB1 or GRUB2

download Netboot kernel and Netboot initrd

$ wget http://ftp.debian.org/debian/dists/jessie/main/installer-amd64/current/images/netboot/debian-installer/amd64/linux
$ wget http://ftp.debian.org/debian/dists/jessie/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz

mount /boot of the target HDD (device /dev/sdX) as root

$ sudo mount /dev/sdX1 /mnt

copy kernel and initrd to mounted /boot of the target HDD

$ sudo cp linux /mnt/netboot-linux
$ sudo cp initrd.gz /mnt/netboot-initrd.gz

unmount /boot of the target HDD

$ sudo umount /mnt

reboot into GRUB command line (press c)
for GRUB1 enter this to boot

root (hd0,0)
kernel /netboot-linux priority=low
initrd /netboot-initrd.gz
boot

for GRUB2 enter this to boot

set root=(hd0,msdos1)
linux /netboot-linux priority=low
initrd /netboot-initrd.gz
boot

http://www.debian.org/releases/stable/installmanual

Remotely with debootstrap

Debian system and similar are so flexible that they can also be installed remotely over SSH on a machine running any version of Linux. But beware, that such procedures are extremely dangerous, fragile, and require expert knowledge. To prevent you from breaking your system, we won’t describe them here.

Installation alternatives for Ubuntu 14.04 LTS

2015-05-15T00:00:00+02:00

Ubuntu 14.04 LTS and similar systems can be installed in many ways, but the alternative Network installer variant is the smallest and cleanest, as it installs directly from the internet. Depending on the selected method and architecture (amd64 is for 64-bit Intel/AMD) you need an appropriate installation image or special boot files.

Locally with network installer

From USB stick

insert an empty USB stick
download a Netboot ISO image

$ wget http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/installer-amd64/current/images/netboot/mini.iso

write image to the USB stick (device /dev/sdX) as root

$ sudo dd if=mini.iso of=/dev/sdX bs=16M; sync

reboot from the USB stick (optionally add kernel parameter priority=low)

From CD/DVD

insert an empty CD or DVD
download a Netboot ISO image

$ wget http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/installer-amd64/current/images/netboot/mini.iso

burn image (file mini.iso) onto the CD or DVD (using K3b or other CD/DVD burning utility)
reboot from the CD/DVD (optionally add kernel parameter priority=low)

From HDD using a special image

insert an empty target HDD
download a HD-media image

$ wget http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/installer-amd64/current/images/hd-media/boot.img.gz

write image to the target HDD (device /dev/sdX) as root

$ gzip -cd boot.img.gz | sudo dd of=/dev/sdX bs=16M; sync

insert the HDD in the target machine
reboot from the HDD (optionally add kernel parameter priority=low)

From HDD using GRUB1 or GRUB2

download Netboot kernel and Netboot initrd

$ wget http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/linux
$ wget http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/initrd.gz

mount /boot of the target HDD (device /dev/sdX) as root

$ sudo mount /dev/sdX1 /mnt

copy kernel and initrd to mounted /boot of the target HDD

$ sudo cp linux /mnt/netboot-linux
$ sudo cp initrd.gz /mnt/netboot-initrd.gz

unmount /boot of the target HDD

$ sudo umount /mnt

reboot into GRUB command line (press c)
for GRUB1 enter this to boot

root (hd0,0)
kernel /netboot-linux priority=low
initrd /netboot-initrd.gz
boot

for GRUB2 enter this to boot

set root=(hd0,msdos1)
linux /netboot-linux priority=low
initrd  /netboot-initrd.gz
boot

Remotely with debootstrap

Windows XP disable system restore

2012-09-18T00:00:00+02:00

One of the features of Windows XP/ME/Vista is a special backup utility called System restore. It automatically creates backups of important files and is used by the operating system to restore files on your computer in case they become damaged. The feature is enabled by default and it creates a hidden folder called _Restore or System Volume Information on each partition where it stores the data (these folders are updated when the computer restarts).

Although this is a desirable functionality, in some cases System restore should be temporarily turned off. A problem appears if a virus, infected file, or other malicious software is backed up. Therefore if you play around too much you can accidentally restore malicious software that will continue to poison your computer. This can happen even if you have a good antivirus program installed, because Windows prevents access to this folder by outside programs. You must disable the System restore utility to remove the infected files.

Windows XP

Right click the My Computer icon on the desktop and select Properties. If you do not have My Computer icon you can accomplish the same thing by opening an application called System which is located in the Control Panel.
Select the System restore tab and a window like the one below should appear.
Put a check mark next to Turn off System restore on All Drives.
Confirm with OK and you will be prompted to restart the computer. Choose Yes.

Windows XP disable system restore

Note: Disabling System restore or reverting to a previously saved restore point could affect your personal data, so create a backup just in case. To re-enable the System restore, follow the above steps, but remove the check mark next to Disable System restore or Turn off System restore on All Drives.

Windows ME

Right click on the My Computer icon on your desktop and select Properties.
Go to the Performance tab.
Click the File System button.
Select the Troubleshooting tab and you should see a window like the one below.
Put a check mark next to Disable System restore.
After you confirm with the OK button you will be prompted to restart the computer. Choose Yes.

Windows ME disable system restore

http://www.wikihow.com/Delete-System-Restore-Files

Windows XP safe mode

2012-09-18T00:00:00+02:00

Sooner or later you will need to start your Windows in Safe mode. Safe mode is a troubleshooting option that starts your computer in a limited state. Only the basic files and drivers necessary to run Windows are loaded and this enables you to fix various problems with device drivers or remove viruses. These instructions below will describe you how to restart in Safe mode on various versions of Windows operating system.

Windows 95
Windows 98/ME
Windows 2000/2003
Windows XP (all editions)
Windows Vista (all editions)

Windows 95

Open up the Start menu and click on Shutdown.
Now select Restart The Computer and confirm it.
Hold down the F8 key on your keyboard as your PC restarts.
If your PC starts beeping then release the key for just a second before holding it down again.
Now sit back and wait for Windows to start up in Safe mode. If Windows doesn’t restart in Safe mode then repeat the process.

Windows 98/ME

Click on the Start button and select Shutdown.
Select Restart The Computer and confirm it.
As the computer restarts, press and hold down the F8 key until the Windows Startup menu appears.
If your PC starts beeping then release the key for a moment and press it again.
Select Safe mode from the Startup menu, and press the Enter button on your keyboard.
If everything went as supposed to, then Windows should start in Safe mode. If Windows doesn’t come up in Safe mode then please try again.

Windows 2000/2003

Open up the Start menu and select Shutdown.
Select Restart and confirm it.
Hold down the F8 key on your keyboard as your PC restarts until the Windows Startup menu appears.
If your PC starts beeping then release the key for just a second before holding it down again.
Select Safe mode from the Startup menu, and press the Enter button on your keyboard.
Now sit back and wait for Windows to start up in Safe mode. If Windows doesn’t restart in Safe mode then repeat the process.

Windows XP

Click on the Start button and select Turn off computer.
Select Restart.
During restart, hold down the F8 key on your keyboard until the Windows Startup menu appears.
If your PC starts beeping then release the key for a moment and press it again.
When the Startup menu appears, use the arrow keys and select Safe mode and press Enter.
If everything went as supposed to, then Windows should start in Safe mode. If Windows doesn’t come up in Safe mode then please try again.

Windows Vista

Click on the Start button, then on the arrow next to Lock button and select Restart.
During restart, hold down the F8 key on your keyboard until the Advanced Boot Options screen appears.
If your PC starts beeping then release the key for a moment and press it again.
When the Advanced Boot Options appears, use the arrow keys and select Safe mode and press Enter.
If everything went as supposed to, then Windows should start in Safe mode. If Windows doesn’t come up in Safe mode then please try again.

http://www.wikihow.com/Get-Safe-Mode-in-Windows-XP

Virus classification

2012-08-08T00:00:00+02:00

Computer threats

An enormous amount of malicious software programs exist, especially for Microsoft Windows operating systems, that endanger your computer or your data in various ways. Depending on the type of behavior they can be classified into a few categories.

Adware: It is designed to display unwanted advertising and sometimes slowdown your computer.
Backdoor: Piece of software that bypasses normal authentication procedures, and allows easier access in the future for the attackers.
Browser hijacker: Specialized software to change your home page and search engine used by web browsers for unwanted advertising or information stealing.
Keylogger: Specialized software capable of recording every keystroke you make on your computer, usually for stealing your passwords or other personal information.
Rootkit: Software packages that try to avoid detection by the user by modifying the operating system. Usually they are deployed with backdoors.
Spyware: A special kind of trojan horse designed to spy on all user actions in all programs. Records keystrokes, passwords, personal information, emails, web surfing activity, and submits harvested information to a malicious user.
Trojan horse: A broad term for malicious programs disguised as something normal or desirable, that attempt to trick the user to willfully install it without realizing its harmful potential.
Virus: A computer virus is a self-replicating program that spreads by inserting copies of itself into other executable code or documents and it behaves similarly to a biological virus.
Worms: A special variant of computer viruses capable of infecting computers over the internet or local networks. Usually their main purpose is to attack certain web sites, send spam, or to act as a trojan horse.
Malware: A catch-all term to refer to any software designed to cause damage to a single computer, server or computer network, whether it’s a virus, spyware, trojan horse… It is a short form for malicious software.

Virus Admilli Service details

2012-08-08T00:00:00+02:00

What does Admilli Service do?

Admilli Service has the ability to install itself automatically while surfing on the internet with Internet Explorer (even under higher security level).

We were unable to determine its exact activity after installation, but it looks like some sort of sophisticated spyware and should not be on your PC.

Antivirus solutions

We tried to detect and clean the virus with the following antivirus and antispyware solutions, that were all up to date (on the 26th December 2004), but none of them found anything!

Therefore we came to the conclusion that the thing is yet unknown to the world and it behaves differently than common viruses (because none of the heuristic detection mechanisms found anything).

More technical results

Admilli Service is a new spyware program that well at least in Windows XP/98 operating system and has the ability to install itself automatically through the Internet Explorer (even under higher security restrictions in many versions of it, also in 6.0 SP2). It forces Internet Explorer to execute some commands which download, copy and install an unsigned add-on on the system. After this we can see that the two new services, called AdmilliServ.exe and AdmilliKeep.exe, are running and from the directory C:\Program Files\Admilli Service\. This two programs have the ability to execute each other after one is closed and therefore it is difficult to close them.

Here are all the things that change on a system after the installation/infection:

New contents in file C:\Windows\setupapi.log:

[2004/12/26 13:50:47 880.74]
#-198 Command line processed: "C:\Program Files\Internet Explorer\iexplore.exe"
#-024 Copying file "C:\DOCUME~1\User\LOCALS~1\Temp\ICD1.tmp\AdmilliServX.dll" to "C:\WINDOWS\Downloaded Program Files\AdmilliServX.dll".
#E361 An unsigned or incorrectly signed file "C:\DOCUME~1\User\LOCALS~1\Temp\ICD1.tmp\AdmilliServX.dll" will be installed (Policy=Ignore). Error 0x800b0100: No signature was present in the subject.

A file was added to C:\WINDOWS\Downloaded Program Files\. There is a new registered key control name and a file that is invisible in Explorer: AdmilliServX.dll (23.040 bytes). The key can be deleted with Explorer, but for the removal of the file you will need to go into MS-DOS console or use another program like Total Commander.

All installed files are placed into C:\Program Files\Admilli Service\. These are: AdmilliComm.dll (60.928 bytes), AdmilliKeep.exe (17.920 bytes), AdmilliServ.exe (26.112 bytes).

New registry entries:

[HKEY_LOCAL_MACHINE\SOFTWARE\Admilli Service]
"param"="84ff9b0589be58f2fbb4f0b2047978d6d2c681f572f44776ea800a2822cf80fd5393a5536ca9d30e8b03:3732336438643833383439636664333333373836306136353164336534633133:Internet%20Explorer:6.0%20SP2%28SV1%29:winxp:flash"
"track"=dword:00000001
"LastUpdate"=dword:41ceb3bc
"reqcount"=dword:00000002

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\App Management\ARPCache\Admilli Service]
"SlowInfoCache"=hex:28,02,00,00,01,00,00,00,00,00,02,00,00,00,00,00,00,58,4d,\
  a0,9e,eb,c4,01,00,00,00,00,44,00,3a,00,5c,00,76,00,69,00,72,00,75,00,73,00,\
  5c,00,41,00,64,00,6d,00,69,00,6c,00,6c,00,69,00,20,00,53,00,65,00,72,00,76,\
  00,69,00,63,00,65,00,5c,00,41,00,64,00,6d,00,69,00,6c,00,6c,00,69,00,4b,00,\
  65,00,65,00,70,00,2e,00,65,00,78,00,65,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00
"Changed"=dword:00000000

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run]
"Admilli Service"="C:\\Program Files\\Admilli Service\\AdmilliServ.exe"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\Admilli Service]
"UninstallString"="C:\\Program Files\\Admilli Service\\AdmilliServ.exe /Remove"
"DisplayName"="Admilli Service"

As you can see the program also added itself to the Add or Remove Programs section in Control Panel. Because the malware came in through Internet Explorer there is still a copy of it in its cache (Temporary Internet Files).

You can also download all files described files (password: virus) and check it yourself.

Removal instructions (the hard way)

You may try some of the antispyware solutions described above (when they are updated) or try the following instruction for removing this malware the hard way:

First of all you will somehow need to deactivate the program. You will need to stop the processes named AdmilliServ* and AdmilliKeep, but this is not as easy as it looks like.
- The easiest method (for Windows XP and NT) to effectively close Admilli Service until next reboot is to press Ctrl+Alt+Del and selecting the Processes tab. There you will just need to end the process tree of the program. To do this right click on AdmilliServ and choose End Process Tree option. Both AdmilliServ and AdmilliKeep should disappear from the processes list and you may continue with instruction 2.
- Another simple method for the deletion is to restart your computer in Safe mode. When you manage to get there none of the suspicious programs are running, so you may continue with instruction 2.
- A trickier method (for Windows XP and NT) is to press Ctrl+Alt+Del and selecting the Processes tab. There you need to lower the priority under which both of the processes are running. This can be done by right clicking on AdmilliServ and then on AdmilliKeep and setting the priority to Low. After that you will somehow need to give your computer some work to do (execute multiple programs really fast) and in the mean time (when your computer is loading all the programs) try to select and end both processes AdmilliServ and AdmilliKeep really fast. If you are lucky and fast enough, you will be able to close both of the programs that won’t reappear again until reboot.
It is also a smart idea to disable the System Restore option during this process.
Locate the directory where Admilli Service installed itself into and delete it with all the files in it. It can usually be found in C:\Program Files\Admilli Service\. With this action you will delete the following files:
- AdmilliServ.exe (26.112 bytes) - main spyware program
- AdmilliKeep.exe (17.920 bytes) - slave program that makes it harder to close the main one
- AdmilliComm.dll (60.928 bytes) - unknown strange dynamic link library
At next you will need to edit your Registry, therefore open a program called Regedit. This can be done by clicking on the Run option in the Start menu and entering regedit.exe inside the text field. You should use this program with care, because invalid or deleted entries may crash your computer and leave it in an unbootable state. Now you will need to locate the following keys on the left side:
- Locate HKEY_LOCAL_MACHINE\SOFTWARE\Admilli Service, select and delete it all by right clicking on it or pressing Del.
- Find the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\App Management\ARPCache\Admilli Service and delete it with all its values (right click on it or press Del).
- Go to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run and select the value called Admilli Service (right click on it or press Del).
Open up the Command Prompt (MS-DOS console) or any other program that allows you to browse through your files except Explorer (for example Total Commander is a good alternative). Locate the directory* C:\WINDOWS\Downloaded Program Files\ and delete the file called AdmilliServX.dll (23.040 bytes).
Locate the same directory again in Explorer and delete a strangely named key associated with AdmilliServX.dll out.
Open up the Control Panel and choose to Add or Remove Programs. Locate Admilli Service in it and click the uninstall button. A window will pop up and complain that some files are missing, but that’s OK, because we removed them earlier.
At the end you may also empty your Temporary Internet Files cache in Internet Explorer. For this you need to select the menu Tools, then Internet options and click on the Delete files button.

Now you can relax, because you are spyware-free or at least free from this Admilli Service virus.

Virus Admilli Service

2012-08-08T00:00:00+02:00

This page is dedicated to the back then yet unknown new virus threat that appeared on many Windows XP/ME/98 computers in January 2005 and was even spreading half a year later! Below are the results of an investigation before any antivirus software was able to remove or even detect it.

What is Admilli Service?

Admilli Service seems to be a adware/spyware/virus threat that has the ability to infect computers running the Windows XP/ME/98 operating system. It can automatically install itself into your PC when you are surfing on the internet with Internet Explorer (even with a higher security level).

After installation it is not yet known what it does… Maybe it logs all your input and collects your passwords, enables hackers to gain access to you computer or use it as a node for mass spamming, tries to infect other computers in you local network… We were unable to determine its exact activity and classification, but it looks like some sort of sophisticated spyware.

(Nowadays it seems that a newer version of Admilli Service is spreading in the wild and it is classified by others as adware/spyware.)

Antivirus solutions

We tried to detect and clean the virus with many different antivirus and antispyware programs (like Symantec Antivirus, Lavasoft Ad-Aware…) that were all up to date (on December 2004), but none of them found anything!

Therefore we came to the conclusion that the thing is yet unknown to the world and it behaves differently than common viruses.

Removal instructions

As nasty as the threat looks like it can be easily removed with a few clicks! On the other hand you may try some of the newest virus removal tools (some detect it already).

The virus or spyware installs itself as a fully legitimate program inside the C:\Program Files directory with registry entries that result in a working uninstall function. So all you need to do is just open up the Control Panel (in Windows XP it can be found under the Start menu) and choose Add or Remove Programs. Locate Admilli Service in the list that comes up and click the Remove (uninstall) button. After the process is finished your computer will be supposedly spyware-free. You may also temporary disable System Restore before doing anything and empty Temporary Internet Files that Internet Explorer stores on your computer (select the menu Tools, then Internet options and click on the Delete files button).

The whole thing can also be removed with the instructions (the hard way).

More technical results

More details about the investigation can be found on the details subpage.

GW labs

Docker Machine with USB support on Windows/macOS

Docker Machine with USB support on Windows/macOS

Related

Docker dovecot-getmail

Usage

Feedback

License

Discourse Sense Classification from Scratch using Focused RNNs

Micro Drone 3.0 Camera API

General

HTTP API

Telnet shell

Serial interface

Related

Discourse Sense Classification from Scratch using Focused RNNs

Conference proceeding

Abstract

Docker debian-cuda

Usage

Host system

Feedback

License

Docker keras-full

Usage

Feedback

License

Docker keras

Usage

Feedback

License

Docker periodic-sync

Usage

Feedback

License

Docker Compose organized naming convention

Install Docker Compose

Related

Organized naming convention

Project development lifecycle

Docker installation on Debian 8

Install Docker Engine

Journald logging driver

Overlay storage driver

Configure systemd unit

Continue

Related

Multiple Docker hosts with socat tunnels

Socat tunnels to Docker hosts

Weave network driver on Debian 8

Weave network driver

Configure systemd service

Related

Simulation rs-skip-gram-in-myhdl

Usage

Implementation

Testing components

Packing a list of signals to a shadow vector

Unpacking a vector to a list of shadow signals

Feedback

License

Learning Representations for Text-level Discourse Parsing

Docker ubuntu-systemd

Usage

Build

Build ubuntu-systemd

Build debootstrap-minbase.tgz

Feedback

License

Related

Learning Representations for Text-level Discourse Parsing

Conference proceeding

Abstract

Issues and workarounds for Debian 8

Systemd ignores decrypt_derived keyscript

Workaround

Related

Systemd hibernation with encrypted swap partition

Related

Nvidia graphic driver kernel updates

Build `ubuntu-systemd`

Build `debootstrap-minbase.tgz`

Systemd ignores `decrypt_derived` keyscript

Skype error loading `libGL.so.1`

DKMS could not locate `dkms.conf`