diff options
Diffstat (limited to '080_blog/00015_Admin')
8 files changed, 263 insertions, 868 deletions
diff --git a/080_blog/00015_Admin/00005_Miscellanous/index.md b/080_blog/00015_Admin/00005_Miscellanous/index.md deleted file mode 100644 index d247e7e..0000000 --- a/080_blog/00015_Admin/00005_Miscellanous/index.md +++ /dev/null @@ -1,866 +0,0 @@ -Miscellanous Notes -================== - -A collection of short notes on things I wish to remember. Many topics are heavily outdated. - -Backing up and Restoring Docker Volumes ---------------------------------------- - - July 3, 2017 - -One of my first questions, when starting my docker experience was -how to backup and restore docker volumes. - -### On the source host - -First of all identify your volumes: - - $ docker ps - $ docker inspect my_container_name | grep -C 1 -i Source - -Now you can backup the directory to a tar on your host machine: - - $ docker run –rm –volumes-from my_container_name -v $(pwd):/backup ubuntu tar cvf /backup/backup_db.tar /var/lib/mysql - -### On the target host - -In order to securely copy and restore the volume on the target host one can run: - - $ scp source_host:~/backup_db.tar . - $ docker run –rm –volumes-frommy_container_name -v $(pwd):/backup ubuntu bash -c “cd /var/lib/mysql && tar xvf /backup/backup_db.tar –strip 1″ - -A Growing Collection of Linux Command Line One-Liners ------------------------------------------------------- - -Please believe me... this collection was really supposed to grow over time... - -inside a direcotry show disk usage of all hidden files and directories and sort by size: - - $ du $(ls .* -d | tail -n +3) -hs |sort -h - -inside a direcotry show disk usage of all files and directories (also hidden) and sort by size. -Exclude ./DATA file. - - $ du . -a -d 1 -h –exclude=’./DATA’ | sort -h - -Tar all files in current directory, excluding ./DATA and ./.cache - - $ tar –exclude=’./.cache’ –exclude=’./DATA’ -cvf home_miguel_20180216.tar . - -Find files in ./ARCHIVE NOT belonging to a specific user: miguel - - $ find ARCHIVE/ \! -user miguel - -set folder/ permissions to Read/Browse only for owner recursively - - $ sudo chmod -R u=r,g=,o= folder/ - $ chmod -R u=rX,g=,o= folder/ - -find all mails from Boban when in the maildir full of mailboxes and print only short headers without bodies: - - $ grepmail -H -B -Y ‘(^TO:|^From:)’ Boban * - -not really a one-lier but will print 256 colors in a bash: - - for i in {0..255} ; do - printf "\x1b[48;5;%sm%3d\e[0m " "$i" "$i" - if (( i == 15 )) || (( i > 15 )) && (( (i-15) % 6 == 0 )); then - printf "\n"; - fi - done - -Some Tools of Choice --------------------- - -### Systems - -* joomla -* redaxo -* typo3 -* wordpress -* mediawiki - -* oscommerce -* opencart - -* owncloud -* alfresco -* mantis - -* piwik / matomo -* loganalyzer -* goaccess -* nagios / icinga - -### Servers - -* apache -* nginx - -* mariadb / mysql -* postfix -* postgis - -* geoserver - -<!-- -## Tools / Servers - openlayers - git / gitweb / stagit / gitolite - xmpp - mumble - docker / registry - kvm - kubernetes - postfix/dovecot / rouncube/postfixadmin - symfony - bootstrap - webGL - websockets - ajax - node/angular - jmeter - ceph ---> - -Compilation Notes ------------------ - -### build your own webkit - - March 14, 2018 - -Let’s compile a release with debug info and install to /usr/local - -~~~~~~ {.bash} -wget https://webkitgtk.org/releases/webkitgtk-2.20.0.tar.xz -tar -xvf webkitgtk-2.20.0.tar.xz -cd webkitgtk-2.20.0 -# install all the libs that will be reported missing in the next step. -# I could not find the woff2 stuff in debian so skipped it... -cmake -DPORT=GTK -DCMAKE_BUILD_TYPE=RelWithDebInfo -DUSE_WOFF2=NO -GNinja -# this takes about 30minutes on my i7-4790K .. zzzzz..zzz -ninja -sudo ninja install -~~~~~~~~~~~ - - pkg-config - - pkg-config uses our new build now: - - [1] https://trac.webkit.org/wiki/BuildingGtk - [2] https://webkitgtk.org/ - -### Build and Install GCC - - April 27, 2015 - -#### Preparations -Read the prequisites at -[https://gcc.gnu.org/install/prerequisites.html](https://gcc.gnu.org/install/prerequisites.html) -and get (most recent versions at time of this writing) the following: - -* gcc (5.2.0) -* binutils (2.25.1) - -#### Configure and Build - -* unpack binutils-x.y.z -* create a new directory binutils-x.y.z-build and inside it run the following commands: - -~~~~~~ {.bash} - $ ../binutils-x.y.z/configure --disable-nls --with-sysroot --enable-targets=all - $ make -j4 - $ make install -~~~~~~~~~~~~ - -* unpack gcc-x.y.z and run the contrib/download_prerequisites script inside. -* create a new directory: gcc-x.y.z-build and inside it run: - -~~~~~~ {.bash} - $ ../gcc-x.y.z/configure --disable-nls --enable-languages=c,c++ --enable-threads - $ make -j4 - $ make install -~~~~~~~~~~~~ - -#### Reference - [1] binutils and gcc README files. - [2] https://gcc.gnu.org/install/ - [3] http://wiki.osdev.org/Building_GCC - [4] http://stackoverflow.com/questions/1726042/recipe-for-compiling-binutils-gcc-together - -monad transformers in action ----------------------------- - January 1,2018 - - *Main Control.Monad.Writer Control.Monad.State> runState (runWriterT (get >>= \a -> tell ["foo"] >> put (a*a) >> tell ["bar"] >> tell [show a])) 5 - -emscripten ----------- - October 1,2017 - - sdl2 port - https://github.com/juj/emsdk - android-ndk-r15c - - * accomplish build sys with: sdl2+input+audio+opengl+SDL_net+SDL_thread @ linux, win, osx, ios, android, win-phone, steam, rasp - -LUKS container over sshfs -------------------------- - July 5, 2017 - -* enable user\_allow\_other in /etc/fuse.conf -* sshfs -o allow\_root user@server:/BACKUPS/ ~/mnt/ -* dd if=/dev/urandom of=~/mnt/megaloman bs=1M count=512 -* sudo cryptsetup -y luksFormat ~/mnt/megaloman -* sudo cryptsetup luksOpen ~/mnt/megaloman vol1 -* sudo mkfs.ext4 /dev/mapper/vol1 -* sudo mount /dev/mapper/vol1 /mnt -* df -h | grep vol1 -* sudo umount /mnt -* sudo cryptsetup luksClose vol1 -* fusermount -u ~/mnt - -NOTE: backup your data example use : rsync -a –info=progress2 source dist - -some areas of interest ----------------------- - December 20, 2014 - -Some General Topics I am reasearching right now (or plan to do it). Or used to reasearch... - -* Neuronal Networks -* Deep Learning -* Random Forrests (and other techniques based on Decision Trees) -* Simmulated Annealing -* (Linear) Integer Programming - - -Inter Process Communication ---------------------------- - March 14, 2018 - -We can attach nicely to same memory segment from 2 different processes: - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.c .numberLines} -// ipc via shared mem -// attach to shared memory; -key_t my_ftok = ftok("~/surf-webext-dom-shared-mem",'a'); - -int mem_seg=shmget(my_ftok,1024*1024,IPC_CREAT|0660); -if(mem_seg==-1) -{ -g_print("shmget failed: %s\n",strerror(errno)); -} - -shared_buf=shmat(mem_seg,NULL,0); -if(shared_buf==(void*)-1) -{ -g_print("shmat failed: %s\n",strerror(errno)); -} -g_print("attached to shared memory.\n"); -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -SSL Certificate CSR -------------------- - - February 20, 2015 - - for apache ssl mod - - first check your old csr if you like: - $ openssl req -text -noout -verify -in CSR.csr - - 1. generate key - $ openssl genrsa -des3 -out www.yourdomain-example.com.key 2048 - - 2. create CSR - $ openssl req -new -key www.yourdomain-example.com.key -out www.yourdomain-example.com.csr - - refs: - [1] https://search.thawte.com/support/ssl-digital-certificates/index?page=content&id=AR1108&actp=LIST - -Competitive Coding ------------------- - March 17, 2015 - -A small selection of websites, which I visit from time to time, addressing competitive coding. - -* topcoder.com -* codeforces.com -* codility.com -* codersclan.ne -* projecteuler.net -* www.codingame.com (need to check this) -* www.codechef.com (need to check this) -* http://psyho.gg/overview-of-programming-contests/ (and read here) -* https://www.hackerrank.com/ -* http://code.google.com/codejam -* https://www.kaggle.com/ – predicitve modelling -* odesk.com / elance - -Comp Science Literature ------------------------ - March 17, 2015 -This is a list of some books, focusing on topics around computer science, which I recently read or am currently reading or want to read :P - -* Meyers, Effective C++ (Addison-Wesley) C++98 only? -* Meyers, More Effective C++ (Addison-Wesley) C++98 only? -* Meyers, Effective STL. C++98 only? -* Sutter, Exceptional C++ -* Sutter, More Exceptional C++ -* Karlson, Beyond the C++ Standard Library: An Introduction to Boost 1st Edition -* Maybe Something on Multiprocessing (?) -* Donald Knuth. The Art Of Computer Programming -* Algorithms 3rd Edition (by Cormen, Leiserson, Rivest, Stein) -* Jonathan Bartlett. Programming from the Ground Up -* C++ in a Nutshell (O’REILLEY, by Lischner) -* C++ Primer (5th Edition) by Lippman, Lajoie, Moo -* The C++ Programming Language 4th Edition -* Anthony Williams, C++ Concurrency in Action: Practical Multihreading -* Dive into Python (2 and 3) by Mark Pilgrim -* Learn You a Haskell for Great Good by Marian Lipovaca -* Real World Haskell -* Version Control with Git (O’REILLEY, by Loeliger & McCullough) (next: 9(10) ?) -* The Linux Command Line, by William Shotts (http://linuxcommand.org/tlcl.php) -* Debian, The Administrators Handbook (by Hertzog and Mas) -* Absolute FreeBSD by Michael W. Lucas -* Modern Operating Systems, Third Edition (by Andrew S. Tannebaum) -* Linux Device Drivers (O’REILLEY, by Corbert, Rubini, Kroah-Hartman) -* Linux Kernel in a Nutshell (O’REILLEY, Greg Kroah-Hartman) - -* Concrete Math (2nd) -* UPENN cis194 - -* STOC '83 Proceedings of the fifteenth annual ACM symposium on Theory of computing / Primitives for the manipulation of general subdivisions and the computation of Voronoi diagrams - -* http://haskellbook.com/ -* Wadler paper "Monads for Functional Programming" -* simon marlow - book (parallel) -* data 6 - course (Github) -* sedgewick & waynes algorithms -* books from appendix, from programming from ground up? - -VNC ---- - April 2, 2015 - -### Server: - $ apt-get install tightvncserver - $ xtightvncserver -geometry 1900×1100 - $ netstat -tap | grep vnc # remember port (e.g.5901) - -### Client: - $ ssh -L 5901:localhost:5901 you@remote # tunnel - $ xtightvncviewer localhost::5901 # in new terminal - -### Kill Server: - $ vncserver -kill :1 # (or :2 etc..) - -Owncloud Notes --------------- - -### Mounting Ownclouds DavFS - - April 15, 2015 - - $ apt-get install davfs2 - - * since owncloud seems to have problems with locks, edit /etc/davfs2/davfs2.conf and set use_locks to 0 in order to allow creating files (as well as probably writing) - - $ mount -t davfs https://secure.sf.com/owncloud/remote.php/webdav /mnt/ - -### rescan/rebuild ownlocud files database - - June 22, 2016 - - inside ownlcoud directory run: - sudo -u www-owncloud php console.php files:scan –all - - -LXC ---- - April 28, 2015 - - maybe worth a try: - - apt-get install lxc; lxc create ctname -t download — -d debian -r jessie - -a amd64; lxc-start -d -n ctname, lxc-attach ctname - - similar tools: chroot/ debootsrap - -Add vim Lang ------------- - May 14, 2015 - - $ sudo apt-get install myspell-pl - $ cd /ush/share/hunspell - $ vim - :mkspell pl pl_PL - -What every programmer should know about ---------------------------------------- - November 7, 2015 - - Undefined behaviour: http://blog.llvm.org/2011/05/what-every-c-programmer-should-know.html - Memory: http://lwn.net/Articles/250967/ - Floating point arithmetic: https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html - -IDE Comparison --------------- - - December 7, 2015 - -While I love working with vim (ctags, taglist, etc..) and the command line, it might be worth to have a look at the following IDE’s for Java and/or C++: - -* IntelliJ IDEA -* Eclipse -* Eclipse CDT -* Codeblocks -* Netbeans -* Codelite -* KDevelop - -Setup Postfix -------------- - -We want to handle different domains on a single postfix server and have virtual users along real unix users. For the virtual users we use uid/gid 5000. - -Relevant config files are: - - /etc/postfix/master.cf - /etc/postfix/main.cf - /etc/postfix/smtpd_sender_login_maps (tells which emails belong to which login) - /etc/postfix/vmailbox (sepcifies our VIRTUAL mailboxes and the names of the spoolfiles) - /etc/postfix/virtual (aliases for our local unix users) - /etc/dovecot/users (specifies the logins, passwords, home directories,..) - /etc/dovecot/conf.d/10-mail (set inbox and mailbox dirs) - -Be careful about the permissions of the different folders, so uid/gid 5000, postfix and dovecot can access as relevant. - -A very nice tutorial on setting up postfixadmin can be found here: - - https://lelutin.ca/posts/installing_postfix_-_clamav_-_spamassassin_-_dovecot_-_postfixadmin_on_debian_squeeze/ - -Windows Specific Notes ----------------------- - -### disable hiberfile.sys - - September 7, 2017 - - run cmd.exe as administrator: - powercfg.exe -h off - -### GTA4 on NVIDIA-GTX 980 - - April 17, 2016 - - Add following flags : -nomemrestrict -norestriction - Otherwise video memory is not identified correctly. - -### Civilization 4 BTS Autosaves - - September 23, 2016 - - In the file: C:\Users\miguel\Documents\My Games\Beyond the Sword\CivilizationIV - - Adjust: - - ; The maximum number of autosaves kept in the directory before being deleted. - MaxAutoSaves = 100 - - ; Specify the number of turns between autoSaves. 0 means no autosave. - AutoSaveInterval = 1 - -Fool’s Wiki Knowledge -===================== - - February 24, 2016 - -Migrated from our former Mediawiki: - -Welcome to the Foolo-Pedia! Foolo-Pedia is a growing collection of interesting, important, ridiculous or useless information gathered and refined by the SoftwareFools team for all Friends of Fools. We decided to mainly keep this Wiki in english to be compatible with all our costumers and friends. Some destinct pages maybe in polish, german or klingon. - -Articles which might be of universal interest, divided by area and topic. - -## Minilinux - - software fools minimal linux - - 1. usb stick - 2. create one big partition (2gb) - 3. mkefs.ext4 /dev/sdx1 - 4. mount /dev/sdx1 /mnt - 5. compile kernel - 6. cp bzImage /mnt - 7. boot - - 8. - grub> search /bzImage - grub> linux /bzImage root=?? - - PROBLEMS WITH kernel config! - -## Kernel Building - - cd linux-source-[xxx] - make mrproper - cp someconfig .config -i - make oldconfig - make menuconfig - make localmodconfig - make localyesconfig - make - (OPT) make modules - su - make install - (OPT) make modules_install - (OPT) update-intiramfs -c -k [kernel-postfix] - - (REMOVE UNWANTED FILES FROM BOOT!) - update-grub - grub-install /dev/sda - reboot - -## GDB - - http://www.cprogramming.com/debuggers.html - - use gcc or clang with : -ggdb -O0 - - h - help - - r - run - c - continue - n - next - s - step - - b - break <line> - p - print <var> - - wa - watch <var> - - bt - frame <number> - list - info locals - - set <var> = <value> - set substitute-path FROM TO - - call <func> - -## clang - - http://llvm.org/releases/3.7.0/docs/CMake.html - http://clang.llvm.org/get_started.html - put MAKEFLAGS="-j8" or similar in front of your CMake invocations. - - extracted llvm to ~/temp/clang/llvm-3.7.0.src/ - extracted clang to ~/temp/clang/llvm-3.7.0.src/tools/clang/ - extracted compiler-rt to ~/temp/clang/llvm-3.7.0.src/projects/compiler-rt - extracted libcxx to ~/temp/clang/llvm-3.7.0.src/projects/libcxx - - $ cd ~/temp/clang/llvm-3.7.0-build - $ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/miguel/opt/llvm-3.7.0 ../llvm-3.7.0.src - $ cmake --build . - $ cmake --build . --target install - -## XMPP - -### Idea - -Today I figured out, once again, that there are many open-source and free (as in freedom) alternatives to all those chat clients and their proprietary protocols, which we dislike and want to avoid. Taking it seriously I decided to set up a XMPP Server on our Softwarefools Server (http://www.softwarefools.com) to finally enable secure and comfortable communication between me, myself and my fellow peers. - -### Starting Point - -* We have a Debian Virtual Private Server (VPS) Running wheezy (stable) -* I just found that tutorial: http://wiki.linuxwall.info/doku.php/en:ressources:articles:ejabberd - -### I. get package - - # apt-get install ejabberd - -### II. adapt config - -In /etc/ejabberd/ejabberd.cfg adapt the hosts line as that: (if you wonder this is Erlang) - - %% Hostname - {hosts, ["softwarefools.com"]}. - -### III. setup certificate - - # mkdir /etc/ejabberd/certs - -For now we skip the rest of that step and proceed with the self-signed certificate provided by Debian. (/etc/ejabberd/ejabberd.pem) - -### IV. restart server - - # /etc/init.d/ejabberd restart - -### V. create user - - # ejabberdctl register miguel softwarefools.com 'pass123' - # ejabberdctl register niki softwarefools.com 'pass321' - -### VI. open ports - - # iptables -I INPUT -p tcp --dport 5222 -j ACCEPT - # iptables -I INPUT -p tcp --dport 5269 -j ACCEPT - -TODO: in fact we need more here! - -### VII. connect with client - -* Get Pidgin http://www.pidgin.im/ -* Connect & Enjoy  - -### Further Work -* certificate -* dns configuration – should provide xmpp info (? - - -## Benchmark - -### Tools for Stress Testing your RIG - - May 12, 2017 - - Some nice tools to stress-test your computer: - - CPU: Prime95 - GPU: FurMark - RAM: MemTest86+ - HDD/SSD: S.M.A.R.T - - Ref: http://www.pcworld.com/article/2028882/keep-it-stable-stupid-how-to-stress-test-your-pc-hardware.html - -### Networking - - 1. Connected my [laptop] and [desktop] via 5meter cat5e cable -> 1000mbit full duplex link - 2. created ~1.5G ramdisks with tmpfs on each. - 3. created a 1 giga file with data from /dev/urandom on [desktop] - 4. copied file with scp from [desktop] to [laptop] - 5. copied file with scp from [laptop] to [desktop] - 6. repeated 4 & 5 10 times : average speed ~70+MB/s from desk & 80+MB/s to desk, link stays up all the time. ping around 0.2-0.3ms - 7. repeated 1-6 with my other 3meter ethernet cable. - 8. tried both direction simultanously which gave a total data throughput even over 100MB/s - -### Hard Disk - -To get some information about the disk run: - -Request identification info directly from the drive, which is displayed in a new expanded format with considerably more detail than with the older -i option. - - $ hdparm -I /dev/sda - -Timings - - $ hdparm -tT /dev/sda - -Clear Cache - - $ echo 3 > /proc/sys/vm/drop_caches - -I am using this block-size and count which results in ~2GB: - - $ dd [...] bs=1048576 count=2048 - -READ: - - $ dd if=test of=/dev/null - $ dd if=/dev/sdX of=/dev/null - -WRTIE: - - $ dd if=/dev/zero of=test - $ dd if=/dev/zero of=/dev/sdX - -smartmontools: - - seatage: - Raw_Read_Error_Rate - Seek_Error_Rate - - example: - % python - >>> 200009354607 & 0xFFFFFFFF - 2440858991 <---- total number - >>> (200009354607 & 0xFFFF00000000) >> 32 - 46 <--- number of errors - -## Fixing HDD after Free Falling - -So you also dropped your hard-disk and some of your secotrs got damaged, -probably by the head hitting the platter? The following short text outlines, -how I approached fixing my disk after it hit the floor. Of course I would -not rely on this disk for mission critical data anymore, but it -might serve as a scratch/temporarily/redundant space. - -Backup all your data in advance because the disk will almost certainly -require reformatting and repartitoning after follwing this procedure! - -Some Tools/Programms I used: - -* smartctl -* hdparm -* grep - -### smartctl - -Try from fastest to slowest to find faulty sector: replace LBAi\_START,LBA\_END with meaningful values. - - smartctl -t [long|conveyence|short|select,LBA_START-LBA_END] -C /dev/sda - -The following did not work for me. Resulted in ‘interrupted by host’ for some reason. -Probably the disks goes to sleep or something. Issueing some read operations every minute might fix this(?) - - -C to run test in foreground mode! - -### hdparm - -You can bi-search via –read-sector if complete ranges are affected (As in my case) and then fix them in one step: - - for i in {36312851..36312886}; do hdparm --write-sector $i /dev/sda; done - -### Example - -Exemplary session fixing one sector: - -**WARNING: this will destroy your data !** - - 1) # smartctl -a /dev/sda | egrep 'Pend|Real|Offline_Unc' - - 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 4 - 197 Current_Pending_Sector 0x0012 098 098 000 Old_age Always - 47 - 198 Offline_Uncorrectable 0x0010 098 098 000 Old_age Offline - 47 - - 2) # smartctl -t short /dev/sda - - 3) # smartctl -l selftest /dev/sda - - [...] 1 Short offline Completed: read failure 90% 10632 152076520 - - 4) # hdparm --read-sector 152076520 /dev/sda # verification - - [...] reading sector 152076520: FAILED: Input/output error - - 5) # hdparm --write-sector 152076520 /dev/sda - - 6) # smartctl -a /dev/sda | egrep 'Pend|Real|Offline_Unc' - - 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 4 - 197 Current_Pending_Sector 0x0012 098 098 000 Old_age Always - 46 - 198 Offline_Uncorrectable 0x0010 098 098 000 Old_age Offline - 46 - -## Raid and LVM - - - Also refer to: http://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/foreward.html - - Here I will write down some of my considarations about setting up a RAID - and LVM. - - We want to get a performance boost and minimize the ssd wear-out. - Also we will not put there any mission-critical files, because the content - of the ssd will not be mirrored. beside regular backups of course. - We also need to take the limited diskspace into account. - while our hdd’s are 2*2TB we only have a single 250MB sdd. - - So we will be looking at files that are: - - * not updated very often - * read often - * not critical if they get lost. - * not too huge - - So I consider for my usecase - * /bin - * /sbin - * /boot - * /lib{,32,64} - - * /usr (without /usr/local) - - That directories will reside on our raid1 since they contain important files: - * /home/miguel - * /root - * /opt - * /usr/local - - Finally we will mount the following things in ram for superfast and temporary space: - - * /tmp - * /var/log (unless you care about logs) - - What about that directories? - consider speed vs. security vs. drive wear-out - - * /etc - * /var - - Never mount swap on your ssd! - - == extending logical volume and fs == - * https://wiki.archlinux.org/index.php/LVM - - # lvextend -L +20G VolGroup00/lvolhome - - resize fs (ext2, ext3 or ext4) - for ext4 this can be done even without unmounting the device - - # resize2fs /dev// - -## Some RAID Problems on my old Desktop - - https://www.thomas-krenn.com/en/wiki/Mdadm_checkarray - https://plone.lucidsolutions.co.nz/linux/io/ssd-on-nvidia-sata-port-generates-error-eh-in-swncq-mode-and-failed-command-read-fpdma-queued/view - https://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array - - Kernel options: sata_nv.swncq=0 - - [ 8625.896029] ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen - [ 8625.896035] ata5.00: failed command: WRITE DMA EXT - [ 8625.896040] ata5.00: cmd 35/00:08:50:06:b0/00:00:13:00:00/e0 tag 0 dma 4096 out - res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout) - [ 8625.896043] ata5.00: status: { DRDY } - [ 8625.896048] ata5: hard resetting link - [ 8625.896050] ata5: nv: skipping hardreset on occupied port - [ 8626.364038] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) - [ 8631.372026] ata5.00: qc timeout (cmd 0x27) - [ 8631.372031] ata5.00: failed to read native max address (err_mask=0x4) - [ 8631.372033] ata5.00: HPA support seems broken, skipping HPA handling - [ 8631.372035] ata5.00: revalidation failed (errno=-5) - [ 8631.372042] ata5: hard resetting link - [ 8631.372044] ata5: nv: skipping hardreset on occupied port - [ 8631.840045] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) - [ 8631.864140] ata5.00: configured for UDMA/133 - [ 8631.864146] ata5.00: device reported invalid CHS sector 0 - [ 8631.864154] ata5: EH complete - [ 8662.888029] ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen - [ 8662.888036] ata5.00: failed command: WRITE DMA EXT - [ 8662.888041] ata5.00: cmd 35/00:08:50:06:b0/00:00:13:00:00/e0 tag 0 dma 4096 out - res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout) - [ 8662.888043] ata5.00: status: { DRDY } - [ 8662.888049] ata5: hard resetting link - [ 8662.888051] ata5: nv: skipping hardreset on occupied port - [ 8663.356038] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) - [ 8663.396135] ata5.00: configured for UDMA/133 - [ 8663.396141] ata5.00: device reported invalid CHS sector 0 - [ 8663.396151] ata5: EH complete - [ 8693.864031] ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen - [ 8693.864038] ata5.00: failed command: WRITE DMA EXT - [ 8693.864043] ata5.00: cmd 35/00:08:50:06:b0/00:00:13:00:00/e0 tag 0 dma 4096 out - res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout) - [ 8693.864045] ata5.00: status: { DRDY } - [ 8693.864051] ata5: hard resetting link - [ 8693.864052] ata5: nv: skipping hardreset on occupied port - [ 8694.332036] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) - [ 8694.356134] ata5.00: configured for UDMA/133 - [ 8694.356141] ata5.00: device reported invalid CHS sector 0 - [ 8694.356150] ata5: EH complete - [ 8724.840030] ata5: limiting SATA link speed to 1.5 Gbps - - - diff --git a/080_blog/00015_Admin/00005_Notes-on-Hosting/index.md b/080_blog/00015_Admin/00005_Notes-on-Hosting/index.md new file mode 100644 index 0000000..1345bf4 --- /dev/null +++ b/080_blog/00015_Admin/00005_Notes-on-Hosting/index.md @@ -0,0 +1,53 @@ +# Miscellanous Notes on Server Hosting specifics + +## LUKS container over sshfs + +* enable user\_allow\_other in /etc/fuse.conf +* sshfs -o allow\_root user@server:/BACKUPS/ ~/mnt/ +* dd if=/dev/urandom of=~/mnt/megaloman bs=1M count=512 +* sudo cryptsetup -y luksFormat ~/mnt/megaloman +* sudo cryptsetup luksOpen ~/mnt/megaloman vol1 +* sudo mkfs.ext4 /dev/mapper/vol1 +* sudo mount /dev/mapper/vol1 /mnt +* df -h | grep vol1 +* sudo umount /mnt +* sudo cryptsetup luksClose vol1 +* fusermount -u ~/mnt + +NOTE: backup your data example use : rsync -a –info=progress2 source dist + +## Ownlcoud + +### Mounting Ownclouds DavFS + + $ apt-get install davfs2 + + * since owncloud seems to have problems with locks, edit /etc/davfs2/davfs2.conf and set use_locks to 0 in order to allow creating files (as well as probably writing) + + $ mount -t davfs https://secure.sf.com/owncloud/remote.php/webdav /mnt/ + +### rescan/rebuild ownlocud files database + + inside ownlcoud directory run: + sudo -u www-owncloud php console.php files:scan –all + +## Postfix + +We want to handle different domains on a single postfix server and have virtual users along real unix users. +For the virtual users we use uid/gid 5000. + +Relevant config files are: + + /etc/postfix/master.cf + /etc/postfix/main.cf + /etc/postfix/smtpd_sender_login_maps (tells which emails belong to which login) + /etc/postfix/vmailbox (sepcifies our VIRTUAL mailboxes and the names of the spoolfiles) + /etc/postfix/virtual (aliases for our local unix users) + /etc/dovecot/users (specifies the logins, passwords, home directories,..) + /etc/dovecot/conf.d/10-mail (set inbox and mailbox dirs) + +Be careful about the permissions of the different folders, so uid/gid 5000, postfix and dovecot can access as relevant. + +A very nice tutorial on setting up postfixadmin can be found here: + + https://lelutin.ca/posts/installing_postfix_-_clamav_-_spamassassin_-_dovecot_-_postfixadmin_on_debian_squeeze/ diff --git a/080_blog/00015_Admin/00008_Docker/index.md b/080_blog/00015_Admin/00008_Docker/index.md new file mode 100644 index 0000000..b7bc5c0 --- /dev/null +++ b/080_blog/00015_Admin/00008_Docker/index.md @@ -0,0 +1,14 @@ +# Docker Volume Backup + +Identify volume: + + $ docker inspect my_container_name | grep -C 1 -i Source + +Backup the directory to a tar file: + + $ docker run –rm –volumes-from my_container_name -v $(pwd):/backup ubuntu tar cvf /backup/backup_db.tar /var/lib/mysql + +Restore the volume: + + $ docker run –rm –volumes-from my_container_name -v $(pwd):/backup ubuntu bash -c "cd /var/lib/mysql && tar xvf /backup/backup_db.tar –strip 1" + diff --git a/080_blog/00015_Admin/00010_Various-Cheat-Sheets/00120_GDB/index.md b/080_blog/00015_Admin/00010_Various-Cheat-Sheets/00120_GDB/index.md new file mode 100644 index 0000000..5aebdf3 --- /dev/null +++ b/080_blog/00015_Admin/00010_Various-Cheat-Sheets/00120_GDB/index.md @@ -0,0 +1,29 @@ +# GDB + + http://www.cprogramming.com/debuggers.html + + use gcc or clang with : -ggdb -O0 + + h - help + + r - run + c - continue + n - next + s - step + + b - break <line> + p - print <var> + + wa - watch <var> + + bt + frame <number> + list + info locals + + set <var> = <value> + set substitute-path FROM TO + + call <func> + + diff --git a/080_blog/00015_Admin/00020_Benchmarking-and-Stress-testing/index.md b/080_blog/00015_Admin/00020_Benchmarking-and-Stress-testing/index.md new file mode 100644 index 0000000..facfc17 --- /dev/null +++ b/080_blog/00015_Admin/00020_Benchmarking-and-Stress-testing/index.md @@ -0,0 +1,68 @@ +# Benchmarking + + May 12, 2017 + +## Tools for Stress Testing your RIG + + Some nice tools to stress-test your computer: + + CPU: Prime95 + GPU: FurMark + RAM: MemTest86+ + HDD/SSD: S.M.A.R.T + + Ref: http://www.pcworld.com/article/2028882/keep-it-stable-stupid-how-to-stress-test-your-pc-hardware.html + +## Networking + + 1. Connected my [laptop] and [desktop] via 5meter cat5e cable -> 1000mbit full duplex link + 2. created ~1.5G ramdisks with tmpfs on each. + 3. created a 1 giga file with data from /dev/urandom on [desktop] + 4. copied file with scp from [desktop] to [laptop] + 5. copied file with scp from [laptop] to [desktop] + 6. repeated 4 & 5 10 times : average speed ~70+MB/s from desk & 80+MB/s to desk, link stays up all the time. ping around 0.2-0.3ms + 7. repeated 1-6 with my other 3meter ethernet cable. + 8. tried both direction simultanously which gave a total data throughput even over 100MB/s + +## Hard Disk + +To get some information about the disk run: + +Request identification info directly from the drive, which is displayed in a new expanded format with considerably more detail than with the older -i option. + + $ hdparm -I /dev/sda + +Timings + + $ hdparm -tT /dev/sda + +Clear Cache + + $ echo 3 > /proc/sys/vm/drop_caches + +I am using this block-size and count which results in ~2GB: + + $ dd [...] bs=1048576 count=2048 + +READ: + + $ dd if=test of=/dev/null + $ dd if=/dev/sdX of=/dev/null + +WRTIE: + + $ dd if=/dev/zero of=test + $ dd if=/dev/zero of=/dev/sdX + +smartmontools: + + seatage: + Raw_Read_Error_Rate + Seek_Error_Rate + + example: + % python + >>> 200009354607 & 0xFFFFFFFF + 2440858991 <---- total number + >>> (200009354607 & 0xFFFF00000000) >> 32 + 46 <--- number of errors diff --git a/080_blog/00015_Admin/00030_Fixing-HDD/index.md b/080_blog/00015_Admin/00030_Fixing-HDD/index.md new file mode 100644 index 0000000..0bae730 --- /dev/null +++ b/080_blog/00015_Admin/00030_Fixing-HDD/index.md @@ -0,0 +1,48 @@ +# Fixing HDD after Free Falling + +So you also dropped your hard-disk and some of your secotrs got damaged, +probably by the head hitting the platter? + +Backup all your data. + +## smartctl + +Try from fastest to slowest to find faulty sector: replace LBAi\_START,LBA\_END with meaningful values. + + smartctl -t [long|conveyence|short|select,LBA_START-LBA_END] -C /dev/sda + +## hdparm + +Bi-search via –read-sector if complete ranges are affected (as in my case) and then fix them in one step: + + for i in {36312851..36312886}; do hdparm --write-sector $i /dev/sda; done + +## Example + +Exemplary session fixing one sector: + +**WARNING: this will destroy your data !** + + 1) # smartctl -a /dev/sda | egrep 'Pend|Real|Offline_Unc' + + 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 4 + 197 Current_Pending_Sector 0x0012 098 098 000 Old_age Always - 47 + 198 Offline_Uncorrectable 0x0010 098 098 000 Old_age Offline - 47 + + 2) # smartctl -t short /dev/sda + + 3) # smartctl -l selftest /dev/sda + + [...] 1 Short offline Completed: read failure 90% 10632 152076520 + + 4) # hdparm --read-sector 152076520 /dev/sda # verification + + [...] reading sector 152076520: FAILED: Input/output error + + 5) # hdparm --write-sector 152076520 /dev/sda + + 6) # smartctl -a /dev/sda | egrep 'Pend|Real|Offline_Unc' + + 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 4 + 197 Current_Pending_Sector 0x0012 098 098 000 Old_age Always - 46 + 198 Offline_Uncorrectable 0x0010 098 098 000 Old_age Offline - 46 diff --git a/080_blog/00015_Admin/00040_Oneliners/index.md b/080_blog/00015_Admin/00040_Oneliners/index.md new file mode 100644 index 0000000..c16a515 --- /dev/null +++ b/080_blog/00015_Admin/00040_Oneliners/index.md @@ -0,0 +1,41 @@ +# Oneliners + +A Growing Collection of Linux Command Line One-Liners + +Please believe me... this collection was really supposed to grow over time... + +inside a direcotry show disk usage of all hidden files and directories and sort by size: + + $ du $(ls .* -d | tail -n +3) -hs |sort -h + +inside a direcotry show disk usage of all files and directories (also hidden) and sort by size. +Exclude 'garbage' file. + + $ du . -a -d 1 -h --exclude=garbage | sort -h + +Tar all files in current directory, excluding ./DATA and ./.cache + + $ tar --exclude=.cache -cvf home_miguel_20180216.tar . + +Find files in ./ARCHIVE NOT belonging to a specific user: miguel + + $ find ARCHIVE/ \! -user miguel + +set folder/ permissions to Read/Browse only for owner recursively + + $ sudo chmod -R u=r,g=,o= folder/ + $ chmod -R u=rX,g=,o= folder/ + +find all mails from Boban when in the maildir full of mailboxes and print only short headers without bodies: + + $ grepmail -H -B -Y ‘(^TO:|^From:)’ Boban * + +not really a one-lier but will print 256 colors in a bash: + + for i in {0..255} ; do + printf "\x1b[48;5;%sm%3d\e[0m " "$i" "$i" + if (( i == 15 )) || (( i > 15 )) && (( (i-15) % 6 == 0 )); then + printf "\n"; + fi + done + diff --git a/080_blog/00015_Admin/00090_Miguels-KVM-Adventures/index.md b/080_blog/00015_Admin/00090_Miguels-KVM-Adventures/index.md index 7f4e105..722d20f 100644 --- a/080_blog/00015_Admin/00090_Miguels-KVM-Adventures/index.md +++ b/080_blog/00015_Admin/00090_Miguels-KVM-Adventures/index.md @@ -194,7 +194,7 @@ A common use case is to run docker inside the virtual guests, which makes it an } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Optionally you can tell rsyslog to log deamon.* entries into a separate file and adjust logrotation as outlined here [5]. +Optionally you can tell rsyslog to log deamon.\* entries into a separate file and adjust logrotation as outlined here [5]. ## Miscellaneous @@ -202,6 +202,14 @@ Optionally you can tell rsyslog to log deamon.* entries into a separate file and virt-host-validate # validate host virtualization setup ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Use this RedHat virtio drivers when you install win10 [6]. + +* https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/ + +Forward ipv4 to an ipv6 only: + +socat TCP4-LISTEN:51247,fork,su=nobody TCP6:[2a01:4f8:192:5112::6]:51247 + ## References [1] https://libvirt.org/formatdomain.html @@ -209,4 +217,4 @@ Optionally you can tell rsyslog to log deamon.* entries into a separate file and [3] http://repo.firewall-services.com/misc/virt/virt-backup.pl [4] https://www.linux-kvm.org/page/Tuning_KVM [5] https://www.wolfe.id.au/2015/05/03/syslog-logging-driver-for-docker/ - + [6] https://www.funtoo.org/Windows_10_Virtualization_with_KVM |