Programming on Raveen Kumar

AWK

Sat, 18 Feb 2023 00:00:00 +0000

Example 1

awk -F ',' '{print $NF}' {{filename}}

To print only rows with columns matching a specific string

awk '$3 == "hello"' file

This will match with, 1 2 hello something but not with, 1 2 world something

Example 2: Awking and cutting some stuff

Lets assume that you want to print the 1st and 3rd field in one line.

echo "A:B:C:D:E:F:G" | awk -F ':' 'ORS=" " {print $1} {print $3}'

Here ORS stands for, The output record separator, by default its a newline.

Cat

Sat, 18 Feb 2023 00:00:00 +0000

Join multimedia files

cat movie.mpeg.0* > movie.mpeg
cat *mp3 > combined.mp3

Coding Guidelines

Sat, 18 Feb 2023 00:00:00 +0000

First rule of coding, do not code
Second rule of coding, write less code whenever you can

cp

Sat, 18 Feb 2023 00:00:00 +0000

Dereference / follow and copy symbolic links

cp -L source dest

COPY ONLY FILE THAT DO NOT EXIST IN THE DESTINATION DIRECTORY - UPDATE

cp -u *.html destination

Cut

Sat, 18 Feb 2023 00:00:00 +0000

Cut from 25th column

https://superuser.com/questions/316785/is-there-a-diff-utility-that-allows-you-to-exclude-columns

cut -c25- file.txt

Cut fields

echo "1:2:3" | cut -d ":" -f 1

1

du

Sat, 18 Feb 2023 00:00:00 +0000

Find size of all files and directories under current directory and sort it based on size

du -sh * | sort -h

FFMPEG notes

Sat, 18 Feb 2023 00:00:00 +0000

Basic usage

ffmpeg -i input.mp4 output.mp4

Scale video to height 1080p

ffmpeg -i input.mp4 -vf "scale=-2:1080" output.mp4

Remove audio track from a video file

ffmpeg -i input.mp4 -an output.mp4

Slow down a video

https://trac.ffmpeg.org/wiki/How%20to%20speed%20up%20/%20slow%20down%20a%20video

ffmpeg -i "$infile" -filter:v "setpts=2.0*PTS" "slowmo-$infile"

Interpolate videos to high fps

ffmpeg -i input.mkv -filter:v "minterpolate='mi_mode=mci:mc_mode=aobmc:vsbmc=1:fps=120" output.mkv

Concat video files

# mylist.txt file format
# this is a comment
file '/path/to/file1.mp4'
file '/path/to/file2.mp4'
file '/path/to/file3.mp4'

ffmpeg -f concat -safe 0 -i mylist.txt -c copy output.mp4

Cut video files

Following example cuts a video from 5 seconds to 10 seconds and exports it. Alternatively, use this software

Locate

Sat, 18 Feb 2023 00:00:00 +0000

Description

Locate is a alternative to find for files. Locate uses updatedb command to update its database. Note that we must run the updatedb command everytime there is file system change.

Example

Create a database

updatedb

Locate files

locate filename

Create a database locally

updatedb_path="$HOME/.local/locate_db"
updatedb_home="$updatedb_path/home.db"
updatedb -l 0 -o "$updatedb_home" -U ~/

Locate files for locally created database

locate -d "$updatedb_home"

pandoc

Sat, 18 Feb 2023 00:00:00 +0000

https://pandoc.org/MANUAL.html

pandoc -o output.html input.txt

Powershell notes

Sat, 18 Feb 2023 00:00:00 +0000

Winget

To install any program using powershell in similar way to linux use, winget

Example:

winget install pandoc

Sed

Sat, 18 Feb 2023 00:00:00 +0000

To replace text

sed 's/text/replacement_text/g' <file_name>

To replace text and write file

sed -i 's/text/replacement_text/g' <file_name> ** Append to beginning of a file sed -i ‘1i text’ file.txt * back

Find and replace a specific line from all files in current directory matching a specific extension

Here, all .md files have

<a id="org55550fd"></a>

Line in it. The following command deletes those lines from all the files.

find . -iname "*.md" -exec sed -i '/<a id=*/d' {} \;

TMUX

Sat, 18 Feb 2023 00:00:00 +0000

The greatest samurai to ever exist?

TMUX plugins

GitHub - tmux-plugins/tpm: Tmux Plugin Manager

GitHub - tmux-plugins/tmux-urlview: Quickly open any url on your terminal window!

GitHub - tmux-plugins/tmux-resurrect: Persists tmux environment across system restarts.

To move a pane from another window to current window

join-pane -s target-window:target-pane

To use 256 colours in Tmux

256 colours in TMUX

You should set the default terminal in .tmux.conf to be screen-256color,

set -g default-terminal "screen-256color"

Windows WSL Notes

Sat, 18 Feb 2023 00:00:00 +0000

https://thomasward.com/wsl2-x11/

Change password

To change to root user

ubuntu config --default-user root

To change password

passwd, passwd <user_name>

Open Windows GUI applications with Linux command line

mspaint.exe file.png
notepad.exe file.txt

https://dev.to/adityakanekar/upgrading-from-wsl1-to-wsl2-1fl9

bc

Fri, 17 Feb 2023 00:00:00 +0000

bc as a hex calculator

obase=16
ibase=16
9+1

bc fractional

scale=10
1/3

Evaluating expressions from commandline

bc <<< "ibase=10; obase=16; $(grep cru_spare -r | awk -F "," '{print $NF}')"

C++ notes

Fri, 17 Feb 2023 00:00:00 +0000

For loops

for (uint32_t i=0; i<10; i++) {
printf("i = %d", i);
}

Prints

printf(“unsigned = %u”, var1); printf(“signed = %d”, var1);

Functions

https://softwareengineering.stackexchange.com/questions/286490/what-is-the-difference-between-function-and-functionvoid 6.7.6.3 Function declarators (including prototypes) 10 The special case of an unnamed parameter of type void as the only item in the list specifies that the function has no parameters. 14 An identifier list declares only the identifiers of the parameters of the function. An empty list in a function declarator that is part of a definition of that function specifies that the function has no parameters. The empty list in a function declarator that is not part of a definition of that function specifies that no information about the number or types of the parameters is supplied.145)

ctags

Fri, 17 Feb 2023 00:00:00 +0000

Basics

Ctags is a tool that makes it easy to navigate large source code projects. It provides some of the features that you may be used to using in Eclipse or other IDEs, such as the ability to jump from the current source file to definitions of functions and structures in other files.

For full info refer this link, https://courses.cs.washington.edu/courses/cse451/10au/tutorials/tutorial_ctags.html

Commands

cd $REPO_ROOT
ctags -R *

In vim

:! ctag -R *

Add the tags file to vimrc

Diff

Fri, 17 Feb 2023 00:00:00 +0000

To diff 2 files

diff file1 file2

To diff 2 files and view side by side

diff -y file1 file2

Documentation notes

Fri, 17 Feb 2023 00:00:00 +0000

Emacs lisp / elisp

Fri, 17 Feb 2023 00:00:00 +0000

(let ((messages-buffer (get-buffer "*Messages*")))
  (display-buffer-at-bottom messages-buffer
    '(display-buffer-pop-up-window
      (window-height . 0.2))))

Errors and bugs

Fri, 17 Feb 2023 00:00:00 +0000

Removing old name: Device or resource busy

https://unix.stackexchange.com/questions/11238/how-to-get-over-device-or-resource-busy

The tool you want is lsof, which stands for list open files.

It has a lot of options, so check the man page, but if you want to see all open files under a directory:

lsof +D /path

That will recurse through the filesystem under /path, so beware doing it on large directory trees.

Once you know which processes have files open, you can exit those apps, or kill them with the kill(1) command.

General programming notes

Fri, 17 Feb 2023 00:00:00 +0000

File structure

Always try to keep files in one directory, but name them properly
Use multiple directories to dump similar data

json

Cannot use bash variables inside json files

ImageMagick

Fri, 17 Feb 2023 00:00:00 +0000

Adding blur effect to images convert input.jpg -blur 0x8 output.jpg

Convert a series of images to gifs convert -delay 100 -loop 0 image* animation.gif

Javascript

Fri, 17 Feb 2023 00:00:00 +0000

Reading Materials

How do you set up a local testing server? - Learn web development | MDN

https://developer.mozilla.org/en-US/docs/Learn/Getting_started_with_the_web/JavaScript_basics

document.querySelector("html").addEventListener("click", function () {
  alert("Ouch! Stop poking me!");
});

The function we just passed to addEventListener() here is called an anonymous function, because it doesn’t have a name. There’s an alternative way of writing anonymous functions, which we call an arrow function. An arrow function uses () => instead of function ():

document.querySelector("html").addEventListener("click", () => {
  alert("Ouch! Stop poking me!");
});

Less

Fri, 17 Feb 2023 00:00:00 +0000

Usage

less filename

Chop long lines with `less`

less -S filename Note that -S at the end doesnot work

Linux notes general

Fri, 17 Feb 2023 00:00:00 +0000

Commands

cat - contatenate
tac - concatenate in reverse

`column` command

The column utility formats its input into multiple columns.

printf "a:b:c\n1::3\n" | column -t -s ':'

a	b	c
1	3

Note that this output is different from one that is printed in terminal. Strange… Below is the terminal output.

a	b	c
1		3

`uuidgen` universal unique id generator

Generate unique ID’s

`script` Records terminal

man script
man scriptreplay

Example

script --timing=file.tm script.out
# Write some scripts
# Exit with `exit` or `^d`
scriptreplay --timing file.tm --typescript script.out

`reset` Recover a borked terminal

Count number of files in a directory

ls -1 | wc -l

Building from source

./configure
./configure --prefix=$path

Redirections

2>&1
2>&1 > /dev/null
>
>/dev/null
>>

Keymaps

Ctrl-c
Ctrl-d
Ctrl-l

cp

cut

less

linux

seq

paste

pstree

du

rsync

sed

ssh

vifm

zip

Readline

C-x C-e to edit current commant in default text editor
or v in vi mode

`ldd` Print shared libraries

This command can be used to find all the shared library or libraries of a program

Lockfile

Fri, 17 Feb 2023 00:00:00 +0000

`lockfile`

Installation

sudo apt install procmail

Description

man lockfile

Suppose you want to make sure that access to the file “important” is serialised, i.e., no more than one program or shell script should be allowed to access it. For simplicity’s sake, let’s suppose that it is a shell script. In this case you could solve it like this:

...
lockfile important.lock
...
access_"important"_to_your_hearts_content
...
rm -f important.lock
...

Now if all the scripts that access “important” follow this guideline, you will be assured that at most one script will be executing between the lockfile and the rm commands.

paste

Fri, 17 Feb 2023 00:00:00 +0000

print 2 files side by side

paste file1 file2

-d {{delimiter}}
-d''

Programming Concepts

Fri, 17 Feb 2023 00:00:00 +0000

Arrays
Associative arrays
dicts
case
while
for
if
exit codes and returns
advicing functions
Fuzzy matching - Fuzzy Matching with python

Ranger file manager

Fri, 17 Feb 2023 00:00:00 +0000

Managing files from commandline might be cumbersome time to time. To make the job easy there are quit a lot of terminal curses based file managers available like, ranger which uses Vi keybindings.

Installation

pip install ranger-fm

usage

ranger

keybindings

Official pages

https://ranger.github.io/documentation.html https://github.com/ranger/ranger

Tabbed mode

Ctrl-n
<Tab> key to switch between tabs

Show hidden files

Ctrl-h
zh

To cd into current curectory after exit

https://superuser.com/questions/1043806/how-to-exit-the-ranger-file-explorer-back-to-command-prompt-but-keep-the-current

alias ranger='ranger --choosedir=$HOME/.rangerdir; LASTDIR=`cat $HOME/.rangerdir`; cd "$LASTDIR"'

Open Multiple files in vim from ranger

Select all the files with ranger

rsync

Fri, 17 Feb 2023 00:00:00 +0000

Usage

rsync -P source dest

This is the command that I used for syncing this website

# For whatever reason rsync excludes tags directory, so --include used
rsync -av --delete --checksum --copy-links -rPuzhi ./public/ root@raveenkumar.xyz:/var/www/raveenkumar/

Example - Directory mirroring

rsync -PCahzvu --delete $ORIG_PATH/$ORIG_DIR $MIRROR_PATH/.

-P -C: Skip revision control files
-a: Archive 
-h: Human readable
-z: zip
-v: Verbose
-u: Update
--delete: Delete directories and files which  is present in mirror directory but not in original directory.

To exclude

--exclude=”\*/file.txt”

https://superuser.com/questions/555799/how-to-setup-rsync-without-password-with-ssh-on-unix-linux

seq

Fri, 17 Feb 2023 00:00:00 +0000

http://www.compciv.org/topics/bash/variables-and-substitution/

seq 1 5
echo $(seq 1 5)
# Or, to create 5 new directories:
# mkdir $(seq 1 5)

Slurm Job manager

Fri, 17 Feb 2023 00:00:00 +0000

To allocate 4 CPU cores and 8 GB of RAM for an emacs process using the srun command, you can use the following command:

srun --cpus-per-task=4 --mem=8192 emacs

The --cpus-per-task option specifies the number of CPU cores to allocate, and the --mem option specifies the amount of memory in megabytes.

To pass the -nw argument to emacs when using the srun command, you can simply include it after the emacs command in the srun command line. For example:

ssh

Fri, 17 Feb 2023 00:00:00 +0000

Cd to a particular dir after ssh

$ ssh -t ostechnix@192.168.225.52 'cd /home/ostechnix/dir1 ; bash'

To automatically install a ssh public key on a remote machine

ssh-copy-id -i ~/.ssh/id_rsa.pub $USER@$HOSTNAME

To manually install a ssh public key on a remote machine

cat ~/.ssh/id_rsa.pub; # copy the contents of the file; paste into ssh/authorized_keys file;
chmod 600 ~/.ssh/authorized_keys

Tar

Fri, 17 Feb 2023 00:00:00 +0000

Untar

tar -xvf file.tar dir_ext

tar

tar -cf file.tar

To compress with gunzip/gzip

tar -xvfz file.tar dir_ext

Gunzip / gzip

cat file | gzip > file.gzip

Unix find

Fri, 17 Feb 2023 00:00:00 +0000

To exclude files or perform reverse match

find . -not -iname "file"

To perform regex matching

find . -regex ".*file"

To find and print only symbolic links in a directory and its subdirectories

find . -type l -print

Zip

Fri, 17 Feb 2023 00:00:00 +0000

Basic usage

zip -r my_folder.zip my_folder

For max compression

zip -9 -r my_folder.zip my_folder

If you want to split the zip file into multiple parts of a certain size, you can use the -s option. For example, to split the zip file into 1 GB parts, you would type:

zip -s 1g -r my_folder.zip my_folder

This will create files named my_folder.zip, my_folder.z01, my_folder.z02, etc. To unzip them, you need to use the unzip command with the first file only 2:

Bash

Thu, 16 Feb 2023 00:00:00 +0000

Bash

Reading materials

Select statement

select opt in y n; do
	echo $opt
done

This creates a loop where the user can select the option. Use case to break out of the loop.

select opt in y n; do
	echo $opt selected
	case $opt in
		y)
			break 
			;;
		n)
			break 
			;;
		*)
			echo wrong option
			;;
	esac
done

Bash variables

$USER
$HOSTNAME
$PS1

To Find or List all environment variables

env

This is useful when we want to manage the shell variables properly

GIT

Thu, 16 Feb 2023 00:00:00 +0000

Never rebase

Never rebase if you commit frequently. Merge is the correct option. Rebase could miss some changes.

2023-01-08 - Grep for a string or a change in all the commits

Also called git pickaxe This will find any string that ever existed in that repo.

git log  -S <string> -p

Removing large files and sensitive files from repo.

I accidentally uploaded about 1.7 GB of data into GitHub, and GitHub only accepts 100 MB max file size.

GNU Make Notes

Mon, 13 Feb 2023 00:00:00 +0000

GNU Make notes

https://www.cmcrossroads.com/article/basics-getting-environment-variables-gnu-make

exit script with exit 0 or make will interpret it as error if there is no 0 returned. Context below

c - Make: *** [] Error 1 - Stack Overflow

How to shoot yourself in the foot

Tue, 07 Feb 2023 00:00:00 +0000

How to shoot yourself in the foot

Described below are several methods for shooting yourself in the foot using various programming techniques.

C
You shoot yourself in the foot.

C++
You accidentally create a dozen instances of yourself and shoot them all in the foot. Providing emergency medical care is impossible since you can’t tell which are bitwise copies and which are just pointing at others and saying, “That’s me over there.”

Hugo Notes

Wed, 01 Feb 2023 00:00:00 +0000

Hugo notes

Tryig to transition to hugo for my website. This link has been hugely helpful.

https://cloudcannon.com/community/learn/hugo-beginner-tutorial/layouts-in-hugo/

Note that Hugo does not publish posts that is on a future date and todays date. It only serves / publishes those are yesterday and before.

Sort and print all tags by count

https://code.luasoftware.com/tutorials/hugo/hugo-list-tags-sort-by-count

Centering images in hugo

http://www.ebadf.net/2016/10/19/centering-images-in-hugo/

Create tags at the end of a blog post

https://www.jakewiesler.com/blog/hugo-taxonomies

Some notes on using Hugo to publish websites

Mercurial

Mon, 01 Jan 0001 00:00:00 +0000

Log

hg log -u USERNAME 
hg log <branch name> -G # -G for graph view

To ignore specific files `.hgignore`

# .hgignore
dumps\*
tmp\*

Branches

hg log -b
hg branch <Branch name> # current working area as a new branch

Patching

hg diff > local_changes.patch
hg import --no-commit local_changes.patch

Updating

hg up
hg clean

Reverting

hg revert <file>
hg revert --all

Check all subrepos

``` bash hg st -S hg diff -S ```

Hg relative args

``` bash hg st –rev -2 ```

Octave

Mon, 01 Jan 0001 00:00:00 +0000

General

;{.verbatim} semicolon at the end of a command supresses the

output of the command

typeinfo{.verbatim} - check data type
close all{.verbatim}
clear{.verbatim}
clc{.verbatim}

Plot


view(azimuth, elevation)

view(90,0) - To view 3d plot as 2d

view(-90,0)

% Octave Legend: <https://octave.sourceforge.io/octave/function/legend.html>

File handling operations


data=load('file_name.mat') - To load a mat file

save("-text", "data.txt", "data") - To save a variable to a file

save data

csvwrite (filename, x) - Write the numeric matrix x to the file filename in comma-separated-value (CSV) format.

data=[struct2cell(load('data.mat')){:}]; - convert a structure to array

c=exist(name); % [check for file/var exists](https://octave.sourceforge.io/octave/function/exist.html)

Reading a binary file


fid=fopen("file.bin", "rb");

data=fread(fid, "uint64");

Loading file into a variable


data=load("file.mat")

Save to file in plain text


dlmwrite("file.txt", data, ",")

csvwrite("file.csv", data)

Structures


data_main.name % will get data from the data_main structure, with object "name"

String manipulation


strrep("String", "i", "1"); % i will be replaced with 1; Str1ng

strcat("String1", "String2"); % Output = String1String2

str=int2str(i) % Integer to string

num=str2num(j) % String to number

Matrix manipulation

Find sizes


ndims(matrix)

rows(matrix)

columns(matrix)

Rearranging - flip


flipr(matrix) % 12...9 to 98...1

Rearranging - permute


%% Converting x, y and z to z, x, y

data.element=permute(data.element, [3 1 2]);

size(data.rdc1);

Sum all elements of a matirx


sum(matrix)

Multiply all elements in a matrix with itself


matrix.*matrix

Concatenation

Matrix

hcat

``` octave

Programming on Raveen Kumar

AWK

Example 1

Example 2: Awking and cutting some stuff

Cat

Join multimedia files

Coding Guidelines

cp

Dereference / follow and copy symbolic links

COPY ONLY FILE THAT DO NOT EXIST IN THE DESTINATION DIRECTORY - UPDATE

Cut

Cut from 25th column

Cut fields

du

Find size of all files and directories under current directory and sort it based on size

FFMPEG notes

Basic usage

Scale video to height 1080p

Remove audio track from a video file

Slow down a video

Interpolate videos to high fps

Concat video files

Cut video files

Locate

Description

Example

Create a database

Locate files

Create a database locally

Locate files for locally created database

pandoc

Powershell notes

Winget

Sed

To replace text

To replace text and write file

Find and replace a specific line from all files in current directory matching a specific extension

TMUX

TMUX plugins

To move a pane from another window to current window

To use 256 colours in Tmux

Windows WSL Notes

Change password

To change to root user

To change password

Open Windows GUI applications with Linux command line

bc

bc as a hex calculator

bc fractional

Evaluating expressions from commandline

C++ notes

For loops

Prints

Functions

ctags

Basics

Commands

Diff

To diff 2 files

To diff 2 files and view side by side

Documentation notes

Emacs lisp / elisp

Open messages buffer in a popup window below

Errors and bugs

Removing old name: Device or resource busy

General programming notes

File structure

json

ImageMagick

Javascript

Reading Materials

Less

Usage

Chop long lines with less

Linux notes general

Commands

column command

uuidgen universal unique id generator

script Records terminal

Example

Chop long lines with `less`

`column` command

`uuidgen` universal unique id generator

`script` Records terminal

`reset` Recover a borked terminal

`ldd` Print shared libraries

`lockfile`