CLI workshop¶
CARTO Solutions Technical Workshops
Jorge Sanz · May 2019
http://bit.ly/1905-cli-workshop
This workshop is intended to show why using a command line interface can be useful on many of our everyday tasks, and more convenient and effective than other graphical interfaces.
Contents¶
- Intro
- Some basic commands
- Viewing and finding things
- Accessing APIs and web services
- Handling local data
- Miscellany
CLI versus GUI¶
| _ | CLI | GUI |
|---|---|---|
| Eease | ❌ | ✅ |
| Functionality | ✅ | ❌ |
| Speed | ✅ | ❌ |
| Multitasking | ❌ | ✅ |
| Automation | ✅ | ❌ |
When to use the command line¶
- Repeated tasks
- Greater control over functionality
- Dealing with lots of information
- Focus in the process
- Accessing remote servers
Piping and redirections¶
In [40]:
# pipe the result of a command into another
cat /etc/hosts | grep localhost
In [41]:
# write the contents of a command results into a file (overwriting)
cat /etc/hosts > /tmp/myhosts
Basic commands¶
- Remember you can use the arrow keys to browse your history.
- The
HISTSIZEenvironment variable defines the size of your history. Ctrl + rallows you to search on your history!!executes the previous commandCTL + Lclean the screen (same as executingclearcommand)CTL + A,CTL + E: go to the begin/end of the lineCTL + W,CTL + U: remove the previous word or the everything before the cursorfcwill open an editor with the last command, useful for editing long ones!$will replace by the last parameter of your last command
In [42]:
# Show system info
uname -a
In [43]:
# Show the current user
whoami
In [44]:
# Show the current directory
pwd
In [45]:
# Show the current date
date +%F\ %T
In [46]:
# Print a variable
echo $HISTSIZE
In [47]:
# Print disk status
df -h | grep -v loop
In [48]:
# Print usage
du ~/media/carto --max-depth=1 --human-readable
In [50]:
# Find processes
ps aux | grep unclutter
In [52]:
# Send a kill signal to the unclutter processs
kill -9 2162
ps aux | grep unclutter
Scripting¶
You can write pretty complex scripts using bash that include functions, control structures, and so on.
In [53]:
# You can do for loops
for i in {1..5}
do
echo "Welcome $i"
done
In [54]:
# If structures... and much more
if [ $HISTSIZE -gt 500 ]
then
echo "Big History!"
else
echo "Short History"
fi
Viewing and finding¶
wc and cat¶
In [55]:
# Counting lines, words, and characters
wc /etc/hosts
In [56]:
# Usually we only want the lines
wc -l /etc/hosts
In [57]:
# Print the beginning of a file
head -n5 /etc/hosts
In [58]:
# Same with the end
tail -n5 /etc/hosts
In [59]:
# Combining pipes with wc to count how many files are in a folder
ls /var/log/*.log* \
| wc -l
find¶
In [60]:
# Finding all files inside a folder based in their names and types
# we redirect errors to the black hole at /dev/null
find /var/log -type f -name "*log*" 2> /dev/null \
| wc -l
In [61]:
# Finding and deleting
find /tmp/ -name ".DS_Store" -type f -delete 2> /dev/null
In [62]:
# Finding and executing a command (count lines in gitignore files)
find ~/src/carto -name ".gitignore" -type f -exec wc -l {} \;
In [63]:
# Print comments in the hosts file
sed -n -e '/\#.*/p' /etc/hosts
In [64]:
# Remove comments and then remove blank lines
sed -e '/\#.*/d' -e '/^\s*$/d' /etc/hosts
Silver Searcher¶
The Silver Searcher, quick find anything on your code
In [65]:
ag betis ~/src/carto/help
In [66]:
# accessing a simple URL
curl http://numbersapi.com/8/26
In [67]:
# making a POST request with a Content Type header
curl \
--data '{"q":"select user"}' \
--header "content-type:application/json" \
https://jsanz.carto.com/api/v2/sql
In [68]:
# making a HEAD request
curl \
--head \
https://jsanz.carto.com/api/v2/sql?q=select+user
In [69]:
# formatting the result using python3
curl -s \
--data '{"q":"select user"}' \
--header "content-type:application/json" \
https://jsanz.carto.com/api/v2/sql \
| python3 -m json.tool
Handling data¶
In [70]:
# just formatting the output
curl -s \
--data '{"q":"select cartodb_id, scalerank, featurecla, name, worldcity from populated_places limit 5"}' \
--header "content-type:application/json" \
https://jsanz.carto.com/api/v2/sql > /tmp/data.json
cat /tmp/data.json | jq .
In [71]:
# processing the results to show only the names
cat /tmp/data.json \
| jq ".rows[].name"
In [72]:
# generate a new JSON with custom properties
cat /tmp/data.json \
| jq ".rows[] | {id: .cartodb_id, scale: .scalerank, name: .name}"
In [73]:
# combining with other tools
curl -sk -G "${CARTO_API_URL}api/v1/map/named/?api_key=${CARTO_API_KEY}" \
| jq ".template_ids[]" \
| tr -d '"' \
| sort \
| head
In [74]:
cat /tmp/data.json | jq ".rows[]" \
| json2csv > /tmp/data.csv
cat /tmp/data.csv
csvkit¶
https://csvkit.readthedocs.io/en/latest/
csvclean: fix common CSV problemscsvcut: remove columnscsvgrep: filtering datacsvjoin: join two CSVs by a given columncsvjson: generate json or GeoJSONcsvlook: pretty print CSVcsvstat: basic statistics
In [75]:
# convert to CSV and pretty print the table
cat /tmp/data.csv \
| csvlook
In [76]:
cat /tmp/data.csv \
| csvgrep -c 4 -m tree \
| csvlook
In [77]:
cat /tmp/data.csv | csvstat
In [78]:
# ogrinfo presents an overview of a layer from a data source: type, extent, SRS, schema, etc
ogrinfo -summary ~/media/carto/flights.gpkg cartodb-query
In [79]:
# ogr2ogr processes and converts a datasource between different formats
# on this example we generate a new Geopackage only with one geometry from the original dataset
# this is useful to upload to carto only a dataset definition (after importing you delete that single row)
ogr2ogr \
-f GPKG \
-overwrite \
-limit 1 \
-nln flights_first \
/tmp/flights_first.gpkg ~/media/carto/flights.gpkg
In [80]:
ogrinfo -summary /tmp/flights_first.gpkg flights_first \
| head -n7
Miscellany¶
autojump¶
Jump around your file system without traversing folders
https://github.com/wting/autojump
I have aliased the command autojump to the key j because I'm so lazy
duc¶
duc is a cached du command, that is, a tool that builds an index of how your hard disk is being used and helps you understand where your gigabytes are. It comes with cli and graphical interfaces.
In [81]:
duc ls ~/media/carto | head
translate¶
In [82]:
# Translator using several providers
# https://github.com/soimort/translate-shell
trans -brief es:en 'cuando se despertó, el dinosaurio aún estaba allí'
trans -brief es:zh+ja 'cuando se despertó, el dinosaurio aún estaba allí'
pygmentize¶
In [83]:
# Pretty print source code
# http://pygments.org/docs/cmdline/
pygmentize -g /home/jsanz/src/sdks/carto-python/carto/fields.py
one liners¶
# Get the top ten commands from your history
# (you may have to change the column index in my case was $3)
history \
| awk '{CMD[$3]++;count++;}END { for (a in CMD)print CMD[a] " " CMD[a]/count*100 "% " a;}' \
| grep -v "./" \
| column -c3 -s " " -t \
| sort -nr \
| nl \
| head -n10
