Wikimedia Discovery/Command Line Tricks

From mediawiki.org

This is a collection of useful *nix command line tricks and productivity tips that not everyone knows about that could improve your command-line fu. Note that not all versions of *nix are created equal, so some of these may work in some OSes or some shells but not in others. (OTOH, once you know it's possible, you can usually find a way to make it happen in your OS/shell.)

Commands[edit]

Each command here has a brief description, a use case or two, and an example or three if needed.

pushd <dir>, popd, pushd[edit]

  • description: controls a stack of directories. pushd <dir> moves you to <dir> and pushes <dir> onto a stack. popd pops the stack and moves you back to your previous directory. pushd (with no argument) swaps the top two items on the stack.
  • use case: jumping back and forth between two directories (config in <dir1>, running stuff in <dir2>), changing directories one or more times and then changing back without having to keep track of the list of directories (e.g., in a bash script)
  • see also https://en.wikipedia.org/wiki/Pushd_and_popd

`<cmd>` or $(<cmd>)[edit]

  • description: convert the output of a command into command line arguments; $(...) can be embedded—whether that is good or bad is left as an exercise for the reader.
  • example:
    • touch `date "+%m-%d"`.txt or touch $(date "+%m-%d").txt create a file with today's date as part of the filename using date to get today's date
    • vi `!!` use the output of the previous command (say, something complicated involving ls) as command line arguments for vi
    • vi `grep -slR ombudsmenwiki | grep Initial` open all files in this directory tree that contain ombudsmenwiki but do not have Initial in their name.

!!, !<#>, !<cmd>, !$[edit]

  • description:
    • !! repeats the last command, possibly in a new context;
    • !-2 repeats the command two steps back in your history. !-1 is the same as !!.
    • !445 repeats command 445 from the output of history.
    • !ls will re-run your most recent ls command.
    • !$ is the last argument from the previous command
  • example:
    • sudo !! repeat previous command with sudo
    • vi `!!` use the output of the previous command (say, something complicated involving ls) as command line arguments for vi
    • !!;!! repeat the previous command twice (do it several times and you will repeat the command 4 times, 8 times, 16 times, etc.)
    • vi foo/bar/baz.php followed by git add !$ adds the file you were just editing to git. Alternatively inside vim % to refer to current file. ex: :!git add %
    • See also: https://jaysoo.ca/2009/09/16/unix-history-and-bang-commands/

bash scripting[edit]

  • description: shell scripts can be really powerful. Simple loops are super useful...
  • example:
    • for cfg in $(ls *.cfg); do cp $cfg bk/; done copy every .cfg file into directory bk/
    • for cfg in $(ls *.cfg); do cp $x $x.bk; done copy every .cfg file to a .cfg.bk backup in the same directory
  • Note though that parsing ls output can be error prone. What if a file has a space in it's name? Use find with null delimited strings:
    • find *.cfg -print0 | while read -r -d $'\0' line; do cp "$x" "$x.bk"; done
  • But really find is pretty powerful all on its own and will handle oddities like files with spaces without thinking about it
    • find *.cfg -exec cp {} {}.bk \;
  • see http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-7.html

grep -R[edit]

  • description: recursive grep, searches all files in this directory and every directory below it. Not as powerful as find but easier to remember
  • example: vi `grep -slR ombudsmenwiki | grep Initial` open all files in this directory tree that contain ombudsmenwiki but do not have Initial in their name.

typing the hard-to-type[edit]

° 00b0
á 00e1
é 00e9
î 00ee
2013
2014
266b
Δ 0394
2714
274f
2190
2192
² 00b2
³ 00b3
2135
α 00b1
µ 00b5
  • CTRL+v <char> interpret <char> (most usefully for tab or return) as just a character. Also works in vi/vim

tmux and screen[edit]

  • tmux and screen both allow you to create a terminal session that will persist after logging out. tmux also allows you to split screens and do all sorts of other cool stuff. To learn more—use the Google-fu, Luke.

vi(m) shortcuts[edit]

A few favorite shortcuts in vi. No discussion allowed about whether vi or emacs or some other lame editor is best.

  • ciw = replace current word
  • ci( = replace inside parenthesis
  • diw = delete current word
  • xp = swap current character and the next one
  • ~ = toggle case of current character; 10~ toggles ten characters
  • w = move forward one word with a restrictive definition of word (ex: stops on each / of a path)
  • W = move forward one word with a generous definition of word (ex: a whole path is considered a single word)
  • J = remove newline at end of line, merging current and next line
  • } = Move to next blank line. Useful for jumping large distances in source files. { does same backwards.

Vim prettiness (adds symbols for eol, tab and spaces at the end of the line (tralling) that shouldn't be there) (vim command line or .vimrc or equivalent). This can make it easy to recognize mixed tabs/spaces for example:

    set list

    set listchars=eol:⏎,tab:▸·,trail:␠,nbsp:⎵

To turn back off:

    set nolist

Config[edit]

These are cool things you can add to your .bashrc or .bash_profile to make your command line more efficient

alias[edit]

alias lets you rename a command (including command line options and piping commands together).

  • alias q=exit quit with one keystroke
  • alias ll="ls -l" just two keystrokes to get a long directory listing
  • alias h="history | grep" now h mwscript will list only lines in your history that include mwscript
  • alias stats="ssh stat1002.eqiad.wmnet" log into stat1002 much more easily
  • alias stuff="cd /really/long/harDtOType/or-hard-to-remember/path/to/useful/stuff" if you type it more than twice a day, make an alias!
  • alias li='find . -maxdepth 1 -iname'  makes lia case-insensitive version of ls
  • alias freq="sort | uniq -c | sort -n" pipe into freq to get a frequency sorted list of lines; use sort -nr as the last command to get the most frequent item first; e.g., cat list_of_terms | freq
  • alias vo=vi or alias grpe=grep alias a common typo to what you intended
  • alias cdcirrus="cd ~/mediawiki/vagrant/mediawiki/extensions/CirrusSearch" Stop typing out this crazy long path you move into several times a day. Now you can cdc<tab><return> and go there.
  • alias duh='du --max-depth=1 -h' a quick list of subdirectories with sizes.
  • alias df='df -h' How much space is left?

history config[edit]

  • export HISTCONTROL=erasedups Only keep one copy of any give command in your history. Very useful is you repeat the same command or set of commands over and over without filling up your history
  • shopt -s histappend
    • If the histappend shell option is enabled, the lines are appended to the history file, otherwise the history file is over-written.
  • export PROMPT_COMMAND='history -a; history -r' synch up history across all terminals
  • hold on to more history
    • export HISTSIZE=1000000 # Number of lines in history. Arbitrarily large so histfilesize takes over
    • export HISTFILESIZE=1000000 # Number of bytes in history

export LC_ALL=en_US.UTF-8[edit]

Makes lots of stuff work better with Unicode

custom bash prompt (PS1)[edit]

You can add all sorts of useful information into your bash prompt, including emoji, directories, the time, color, and more. Add some version of these commands to your .bashrc or equivalent. It can be useful to have one color/emoji/etc on one (real/virtual) machine and a different config on another, so you can always tell where you are.

SSH config[edit]

See https://wikitech.wikimedia.org/wiki/Managing_multiple_SSH_agents and https://office.wikimedia.org/wiki/Discovery_Analytics#SSH_Configuration

For Mac-specific SSH key config, see https://github.com/jirsbek/SSH-keys-in-macOS-Sierra-keychain

Other Specific Tools[edit]

These are some specific tools that people have configured for themselves. They may or may not be useful, but may also provide inspiration.

up[edit]

A bash function (goes in .bashrc or equivalent) to navigate around deep directory trees:

up() {
    local p= i=${1:-1}
    while ((i--)); do
        p+=../ 
    done
    cd "$p" && pwd
}

Usage: up 3 goes up three directories

Auto-magic tmux into place on first opened console from .bashrc (with irc):[edit]
if [ -z "$TMUX" ]; then
    base_session='my_session'
    if ! tmux has-session -t $base_session; then
        tmux new -d -s $base_session 'irssi' \; split-window  \; attach \;
    fi
fi

Turns on tmux and opens an irc session using irssi on login. tmux is generally awesome and has many cool features.

Navigation[edit]

These are some useful command-line navigation shortcuts.

up-arrow & down-arrow[edit]

Move through previous command line commands; edit or just re-run them

CTRL-r <string>[edit]

Search back through your history for a command matching what you type. CTRL-r mwscript will take you to your last command with mwscript in it. CTRL-r again will jump back to the matching command before that!

emacs-style navigation & cut/paste[edit]

  • CRTL-e: go to end of line
  • CRTL-a: go to beginning of line
  • CRTL-k: cut to the end
  • CTRL-u: cut to the beginning
  • CRTL-w: cut the previous word
  • CRTL-y: paste
  • ALT-f: forward one word
  • ALT-b: backwards one word

See also[edit]