Index
Table of contents
Index
Symbols != (not-equal-to operator), 127
(pound sign), 53
$ (anchor), 237 % (per cent sign), 78 -80 & (set intersection operator), 133 -135
- (asterisk) duplicating lists with, 100 duplicating strings with, 68 duplicating tuples with, 96 exploding/gathering positional arguments with, 147 -149 multiplication operator, 40 wildcard, 230 ** (asterisks) dictionary operator, 122 , 396 exponentiation, 43 keyword arguments with, 149 , 195
operator, 40 , 68, 95, 101
- (difference), 134 . (directory), 268 .. (parent directory), 268 / (slash), 273 <= (subset), 134 , 135 = (assignment operator), 28 , 106, 125 == (equality operator), 57 , 127
(proper superset), 135 = (superset), 135 [key], 119 -121, 121 [offset] (see offset) [] (square brackets), 97 \ (backslash), 54 , 66, 67, 273 \n (new line), 66 \N{name}, 221
\t (tab), 66
\u, 221
\U, 221
_ (underscore), 39 , 163
_ (underscores), 163
__ (double underscores), 152 , 163, 184, 190,
206
__init__ () method, 172
__str__ () method, 192
{ } (curly brackets)
for dictionary creation, 117 -118, 124
for string formatting, 80 -82
ǀ (vertical bar), 134
‸ (anchor), 237
‸ (exclusive or), 134
A
absolute imports, 205
abspath() function, 273
accumulate() function, 213
add() function, 131 , 329
addition, 40
add_all () function, 329
aggregation, 193
algebra, 496
algorithms, data structures and, 443
alias, 202
alignment functions, 77
Amazon Web Services (AWS), 372
Anaconda
conda package manager, 519
installation, 518
anchor ($), 237
animation, 458
581
anonymous functions, 156 Ansible, 369 Apache web server, 386 API (application programming interface), 320 , 360 , 400 append() function, 100 arange() function, 491 arguments, 142 -151 default parameter values for, 146 defined, 143 exploding/gathering keyword arguments, 149 exploding/gathering positional arguments, 147 -149 keyword, 146 keyword-only, 150 mutable and immutable, 151 None, 144 positional, 145 arithmetic calculations, 489 array() function, 491 arrays, 489 (see also NumPy) changing with reshape(), 493 getting elements from, 494 making with arange(), 491 making with array(), 491 making with zeros()/ones()/random(), 493 math functions and, 495 packed sequences with, 489 art (see graphics) ASCII, 220 assembly language, 506 assertion, 421 assignment operator (=), 28 , 106, 125 assignment, copying versus, 33 asterisk (*) duplicating lists with, 100 duplicating strings with, 68 duplicating tuples with, 96 exploding/gathering positional arguments with, 147 -149 multiplication operator, 40 wildcard, 230 asterisks (**) dictionary operator, 122 , 396 exponentiation, 43 keyword arguments with, 149 , 195 asynchronous (term), 284
asynchronous functions, 165
asynchronous tasks, 521 -526
asyncio alternatives, 524 -525
coroutines and event loops, 522 -523
frameworks and servers, 526
versus other approaches, 525
asyncio library
about, 297 , 522
alternatives to, 524 -525
attributes, 171
accessing, 181 -186
class and object, 185
finding with self argument, 180
getters and setters, 181
initialization and, 172
name mangling for privacy, 184
properties for access, 182 -183
properties for computed values, 184
attrs package, 197
audio, 466
Azure, 372
B
back pressure technique, 300
backslash (\), 54 , 66, 67, 273
bare metal, 508
bases, 44
basicConfig() function, 438
BeautifulSoup, 402
Beowulf cluster, 509
best practices
code, testing, 418 -428
debugging, 428 -437
finding code, 410
integrated development environments
(IDEs), 413 -416
logging, 437 -439
optimizing code, 439 -446
resources, 449 -452
source control, 446 -449
big data, 369
binary data, 238 -244
bit-level integer operators, 244
bytes and bytearrays, 238 -240
converting bytes/strings with binascii(), 244
converting with struct, 240 -244
practice exercise answers, 552 -557
practice exercises, 245
reading binary files, 265
582 | Index
third-party tools for, 243 writing binary files, 264 binary files, 317 HDF5, 317 padded files and memory mapping, 317 spreadsheets, 317 TileDB, 318 binascii() function, 244 bit-level integer operators, 244 bits, 505 Blender, 459 Bokeh, 465 bool() function, 38 boolean operators, 57 booleans, 37 -51 Bottle, 389 -392 break statement, 88 , 90 breakpoint() function, 436 breakpoints, 434 business applications, 467 -483 business tasks, 468 data processing, 469 -473 data security, 474 financial tools, 474 government data sources, 473 maps, 475 -482 Microsoft Office Suite, 468 open source packages, 474 practice exercise answers, 573 practice exercises, 483 bytearrays, 238 -240 bytes, 220 , 238-240, 505 BytesIO, 275 -276
C C, 12 C#, 13 C++, 12 caches, 503 caches, in Redis, 339 calc() function, 289 calendars/clocks, 247 -258 challenges of date representation, 247 datetime module, 249 -251 leap years, 248 practice exercise answers, 557 practice exercises, 258 reading/writing dates and times, 253 -257 standard Python time interconversions, 257
third-party modules for, 257
time module, 251 -253
call() function, 279
capitalization, 76
case, 76
centering of string, 77
chain() function, 212
characters, defined, 63
chdir() function, 272
check_output () function, 278
Cheese Shop (Python Package Index ), 410
Chef, 369
chmod() function, 270
chown() function, 270
classes
aggregation and composition, 193
attribute assignment, 185
attributes, 578
class methods, 187
composition, 193
defined, 32
inheritance, 174 -180
method types, 186 -188
methods and, 172
modules and objects versus, 194
practice exercise answers, 546 -550
practice exercises, 198
clear() function, 104 , 124
client-server system, 376
closures, 155
cloud computing, 370 -373, 509
clusters, 509
code, 53 -61
comments, 53
continuation character (\), 54
debugging, 428 -437
distributing, 449
finding, 410
optimizing, 439 -446
source control for, 446 -449
testing, 418 -428
code structures
comparative statements, 55 -58
iterating over, 212
membership operator, 59
practice exercise answers, 530
practice exercises, 61
true/false values, 58
walrus operator, 60
Index | 583
columns, 318 combining strings, 68 command automation, 282 comments, 53 Common Gateway Interface (CGI), 385 comparisons lists, 109 operators for, 57 tuples, 96 complex numbers, 487 composition, 193 comprehensions dictionaries, 128 generator, 115 , 158 lists, 111 -114 sets, 136 computed values, 184 concatenation strings, 68 tuples, 96 concurrency, 284 -301 asynchronous versus other approaches, 525 asyncio module, 297 concurrent.futures module, 289 -292 green threads, 292 -295 practice exercise answers, 560 practice exercises, 301 processes and, 286 queues and, 285 Redis, 297 -300 threads and, 287 -289 twisted framework, 295 concurrent.futures module, 289 -292 conda, 519 configparser module, 316 configuration files, 315 containers, 508 continuation character (), 54 continue statement, 88 , 90 continuous integration, 428 cooperative multitasking, 522 copy() function copying files with, 269 copying keys and values from one dictio‐ nary to another, 125 copying list with, 107 copying, assignment versus, 33 coroutines, 522 -523 count() function, 104
counter() function, 209
CPU (central processing unit), 503
CPU bound, 284
crawling, 401 -403
CRUD (Create-Read-Update-Delete), 320
CSV (comma-separated values) format,
304 -306
curl, 378
curly brackets ({ })
for dictionary creation, 117 -118, 124
for string formatting, 80 -82
cycle() function, 212
Cython, 444
D
Dask, 370
data munging, 469
data processing (business data), 469 -473
extracting/transforming/loading data,
469 -472
validation, 472
data science, 219 , 498
data serialization, 361 -363
data storage (see persistent storage)
data structures
algorithms and, 443
assigning value to multiple variable names,
33
binary data, 238 -244
comparisons, 136
creating complex, 137
literal values, 26
mutability, 25
data types (see types)
data visualization, 461 -465
Bokeh, 465
Seaborn, 464
data wrangling, 469
databases
document databases, 339
full-text, 341
graph databases, 340
NoSQL data stores, 330 -341
relational, 318 -330
terminology, 318
time series databases, 340
web framework support for, 397 -398
dataclasses, 196
DataFrame, 498
584 | Index
dates, representations of, 247 (see also calendars/clocks) datetime object, 249 -251, 311 daylight savings time, 253 DB-API, 320 dbm databases, 330 DDL (data definition language), 319 debugging, 428 -437 best practices, 428 -437 breakpoint() for, 436 decorators for, 429 printing out strings, 429 with pdb, 430 -436 decimals, floating points with, 488 decode() function, 226 decorators, 159 -160, 429 deepcopy() function, 108 defaultdict() function, 208 del statement, 103 delete an item from list with, 102 deleting individual dictionary items, 124 deque, 211 dict() function converting two-value sequences into dic‐ tionaries, 119 creating dictionaries, 118 dictionaries, 117 -129 adding/changing items, 119 -121 assignment versus copying, 125 combining, 122 comparing, 127 comprehensions, 128 converting two-value sequences into, 118 copying, 125 -126 creating, 117 -118 defined, 7 deleting all items, 124 deleting individual items, 124 getting all key-value pairs, 122 getting all keys, 121 getting all values, 122 getting items by [ key ] or with get(), 121 getting length, 122 iterating over, 127 practice exercise answers, 541 -544 practice exercises, 138 sets versus, 129 testing for keys, 125 DictReader() function, 305
dict_keys () function, 122
difference() function, 134
directory operations, 270 -272
(see also pathnames)
changing current, 272
creating, 270
creating subdirectories, 271
definition of, 270
deleting, 271
listing contents of, 271
listing matching files, 272
directory, defined, 259
Disco, 370
distributed computing, 509
division, 41
Django, 397
DML (data manipulation language), 319
Docker, 373
docstrings, 152
doctest, 425
document databases, 339
documentation, 152 , 416
Domain Name System (DNS), 359
double underscore (__), 152 , 163, 184, 190, 206
double() function, 430
duck typing, 188
dunder, 152
(see also double underscores (__))
dynamic languages, 13
E
electricity, basics of, 502
ElementTree module, 308
elif statement, 56
else statement, 55 , 89, 90
email modules, 360
empty set, 129
encode() function, 224
epoch, 251
equality operator (==), 57 , 127
error handling, 165 -168
(see also exceptions)
escape sequences, 64 , 66, 67
Esri, 475
ETL (extract, transform, load), 469 -472
event loops, 522 -523
event-based programming, 293
event-based servers, 388
exceptions
Index | 585
defining exception types, 167 handling errors with try and except, 166 logging error messages, 437 -439 exclusive or (^), 134 exists() function, 268 expiration dates, 339 expire() function, 339 expireat() function, 339 exponentiation (**), 43 Expression Language (SQLAlchemy), 326 extend() function, 101
F f-strings, 64 , 82 false values, 58 , 144 fanin pattern, 350 fanout pattern, 350 FIFO (first in, first out) queue, 103 file handling, 259 -270 (see also directory operations; pathnames) changing file names, 269 changing ownership, 270 changing permissions, 270 checking for existence of files, 268 checking type of, 268 copying files, 269 creating/opening files, 259 deleting files, 270 file operations, 267 -270 getting pathnames, 273 getting symlink pathnames, 273 linking files, 269 memory mapping, 267 pathnames, 272 -275 practice exercise answers, 558 practice exercises, 276 renaming files, 269 file input/output, 259 -267 changing position within a file, 265 -267 closing files automatically, 265 reading binary file with read(), 265 writing binary file with write(), 264 writing text file with print(), 260 writing text file with read(), 262 writing text file with write(), 261 file system, defined, 270 file, defined, 259 file-like object, 275 financial applications, 474
find() function, 75
findall() function, 232
fire-and-forget technique, 300
Flask, 392 -396
flat file, 259 , 303
floating-point division, 41
floats (floating-point numbers), 49 , 488
folder, defined, 259
fonts, Unicode characters and, 227
for loops, 89 -92
canceling, 90
check break use with else, 90
generating number sequences, 91
iterating lists, 109
iteration, 89 -92
format() function, 80 -82
formatting
f-strings, 82
new style: { } and format(), 80 -82
old style (%), 78 -80
strings, 77 -83
fractions, 489
frozenset() function, 136
ftplib module, 360
full-text databases, 341
functions, 141 -168
(see also methods; specific functions)
anonymous, 156
arguments and parameters, 142 -151
as first-class citizens, 152 -154
asynchronous, 165
calling with parenthesis, 142
closures, 155
decorators and, 159 -160
default parameter values, 146
defining, 141
docstrings, 152
elements of, 38
generator functions, 157
inner functions, 154 -156
keyword arguments and, 146
methods and, 172
namespaces and scope, 161 -163
positional arguments and, 145
practice exercise answers, 544 -546
practice exercises, 168
recursion with, 164
586 | Index
G game development, 465 garbage collector, 32 generator comprehensions, 115 , 158 generators, 157 generator comprehensions, 115 , 158 generator functions, 157 geopandas, 479 get() function, 121 getaddrinfo() function, 359 gethostbyname() function, 293 , 359 getoutput() function, 278 getstatusoutput() function, 279 getter methods, 181 gevent library, 292 -295 Git, 446 -449 glob() function, 272 Global Interpreter Lock (GIL), 289 global variables, 161 globals() function, 162 gmtime() function, 252 Go (Golang), 14 Google Cloud Platform, 372 graph databases, 340 graphical user interfaces (GUIs), 459 -461 graphics 2-D, 453 -458 3-D, 458 3-D animation, 458 games, 465 GUIs, 459 -461 plots/graphs/visualization, 459 -461 practice exercise answers, 573 practice exercises, 466 graphs, 461 -465 green threads, 292 -295 gRPC, 368 GTK+, 460
H Hadoop, 369 hard drives (HDD), 503 hard links, 269 hardware, evolution of computing and, 501 -504 hashes, 332 -340 HDF5, 317 higher-level languages, 507 Houdini, 459 HTML (Hypertext Markup Language), 227 , 309
httpbin, 380
httpie, 379
I
I/O bound, 284
IDLE, 413
if statements, 55 -58
ImageMagick, 457
immutability, 25
import statement, 199 -201
imports, relative/absolute, 205
in (membership operator)
iterating lists, 109
iterating over all items in set, 132
multiple comparisons with, 59
test for existence of value in list with, 104
test for value in set with, 132
testing for keys in a dictionary, 125
index() function
find list items offset by value, 104
for finding offset of substring, 75
index, in relational databases, 318
infinite loop, 88
inheritance, 174 -180
adding a method, 176
calling parent method, 177
from parent class, 174
mixins, 180
multiple, 178 -180
overriding a method, 175
initialization, 28 , 172
inner() function, 154 -156, 155
insert() function, 100
installation
Anaconda, 518
conda, 519
package manager, 412
packages, 410 -413
pip, 411 , 517
pipenv, 412
Python 3, 511 -519
SQLAlchemy, 324
virtualenv, 412 , 517
instance methods, 186
instruction set, 505
int() function, 47 -48
integer division, 41
integer overflow, 49
integers, 38 -49
Index | 587
bases, 44 bit-level operators, 244 literal, 38 operators, 40 -41 precedence of mathematical operations, 43 size in Python 2 versus Python 3, 49 type conversions, 47 -48 variables and, 41 -43 integrated development environments (IDEs), 413 -416 IDLE, 413 IPython, 414 Jupyter Notebook, 416 JupyterLab, 416 PyCharm, 413 interactive interpreter, 18 , 65 internet (see World Wide Web) intersection() function, 133 invoke package, 282 IPython, 414 isabs() function, 268 isfile() function, 268 islink() function, 269 isoformat() function, 249 , 250 issubset() function, 134 issuperset() function, 135 items() function, 122 iteration code structures, 212 dictionaries, 127 lists, 109 loops, 89 -92 multiple sequences, 110 sets, 132 tuples, 96 iterators, 89 -92, 157 itertools, 212
J Java, 13 join() function, 73 , 105 JSON (JavaScript Object Notation), 8, 309-312 JSON-RPC, 365 Jupyter Notebook, 416 JupyterLab, 416 justification, 77
K key-value stores (see NoSQL data stores)
keys
copying, 123 , 125
deleting items by, 124
dictionaries and, 117
getting all, 121
getting item by, 124
handling missing, 207 -209
sets and, 129
keys() function, 121
keyword arguments, 146
exploding/gathering, 149
in named tuples, 195
keyword-only arguments, 150
Kivy, 460
Kubernetes, 373 , 510
L
lambda functions, 156
layers, 343
leap years, 248
left justification, 77
len() function
counting key-value pairs in dictionary, 122
counting Unicode characters with, 223
determining string length with, 72
get list length with, 106
getting length of set with, 131
libraries (see specific libraries)
LibreOffice, 468
LIFO (last in, first out) queue, 103
linear algebra, 496
link() function, 269
links, 269
Linux
package manager, 412
Python 3 installation, 517
shell, 12
list comprehension, 111 -114
list() function
copying lists with, 107
creating empty list with, 97
creating/converting lists with, 97
dictionaries and, 122
listdir() function, 271
lists, 97 -114
adding item by offset to, 100
adding item to end, 100
assign values to sublist with slice, 102
assigning to more than one variable, 106
588 | Index
changing item by [offset], 101 combining, 101 comparing, 109 convert to string, 105 copying with copy(), list(), or slice, 107 copying with deepcopy(), 108 count occurrences of a value, 104 create with a comprehension, 111 -114 creating from string, 98 creating with [], 97 creating/converting with list(), 97 defined, 34 delete all items, 104 delete an item by offset with del, 102 delete an item by value, 103 duplicate all items with *, 100 extracting single value from, 98 extracting subsequence from, 99 find items offset by value, 104 get item by offset and delete, 103 get length, 106 in Redis, 335 iterating multiple sequences, 110 iterating with for and in, 109 of lists, 114 practice exercise answers, 537 -540 practice exercises, 115 reordering items, 105 test for existence of value in, 104 tuples versus, 114 literal integers, 38 literal values, 26 load() function, 314 locals() function, 162 localtime() function, 252 logging, 437 -439 logical (boolean) operators, 57 lookup() function, 221 loops, 87 -92 canceling, 88 coroutines and event loops, 522 -523 for loops, 89 -92 iterating lists, 109 iterating over, 89 -92 practice exercise answers, 536 -537 practice exercises, 92 skipping ahead, 88 while loops, 87 -89
M
machine language, 505
macOS
package managers, 412
path separators, 273
Python 3 installation, 515 -516
magic methods, 190 -193
magnetic disks, 503
MapReduce, 369
maps, 475 -482
applications and data, 482
formats, 475
geopandas and, 479
packages, 480
shapefiles and, 476 -478
marshalling (see data serialization)
match() function
exact matches, 231
specifying output, 238
math functions, 485 -487
mathematics, 485 -490
complex numbers, 487
floating points with decimals, 488
math functions, 485 -487
matrix multiplication, 490
operator precedence, 43 , 57, 575
packed sequences with array, 489
Pandas, 498
rational arithmetic with fractions, 489
SciPy, 497
statistics module, 489
matplotlib, 461 -463
matrix multiplication, 490
Maya, 459
membership operator (see in (membership
operator))
memcached server, 331
memory, 503
memory mapping, 267 , 317
Mercurial, 446
MessagePack RPC, 366
methods, 172
adding, 176
class methods, 187
finding with self argument, 180
instance methods, 186
magic methods, 190 -193
method types, 186 -188
overriding, 175
Index | 589
resolution order, 178 static methods, 187 Microsoft Azure, 372 Microsoft Office Suite, 468 mixins, 180 mkdir() function, 270 modules, 199 -202 (see also specific modules) classes and objects versus, 194 importing, 199 -201 importing specific part of, 202 importing with alias, 201 packages and, 206 practice exercise answers, 551 practice exercises, 216 most_common () function, 210 mro() method, 178 multiplication, 40 multiprocessing module, 279 music, 466 mutability, 25 MySQL, 323
N name() function, 221 named tuples, 195 namespace packages, 205 namespaces, 161 -163 ndarray, 490 Netcat, 349 networks/networking, 343 -374 basics, 509 data serialization, 361 -363 Docker, 373 Domain Name System and, 359 email modules, 360 for big data, 369 internet services, 359 patterns for, 350 -359 practice exercise answers, 564 -571 practice exercises, 373 publish-subscribe model, 355 -359 remote management tools, 369 Remote Procedure Calls, 363 -369 request-reply patterns, 350 -355 web services/APIs, 360 NGINX web server, 388 None value, 144 nonvolatile storage, 503
normalization, of Unicode characters, 228
nose, 426
NoSQL data stores, 330 -341
dbm family, 330
document databases, 339
graph databases, 340
memcached, 331
Redis, 332 -340
time series databases, 340
now() function, 250
null set, 129
Numba, 445
numbers, 37 -51
booleans and, 37 -51
floating-point, 49
integers, 38 -49
practice exercise answers, 529
practice exercises, 51
NumPy, 490 -497
array math, 495
arrays with arange() function, 491
arrays with array() function, 491
arrays with zeros()/ones()/random(), 492
change array shape, 493
getting elements from array, 494
linear algebra functions, 496
O
objects
accessing attributes, 181 -186
adding methods, 176
attributes, 171
calling parent methods, 177
class attribute assignment and, 185
class definition for, 170
classes versus, 194
composition, 193
data as, 23
defined, 169
finding attributes/methods, 180
inheritance and, 174 -180
initialization, 172
magic methods, 190 -193
method types, 186 -188
methods and, 172
modules versus, 194
named tuples, 195
overriding methods, 175
packages and, 206
590 | Index
polymorphism in, 188 practice exercise answers, 546 -550 practice exercises, 198 private attributes, 181 self argument, 180 simple, 170 -173 special methods, 190 -193 ODM (Object Data Manager/Object Document Mapper), 339 offset, 6 adding item to list by, 100 changing list item by, 101 character extraction from string, 69 delete a list item by, 102 extracting single value from list with, 98 find() function, 75 getting list item and deleting, 103 index() function, 75 , 104 seek() function and, 265 -267 substring extraction with slice, 70 -72 tell() function, 265 ones() function, 493 opcodes, 505 open() function, 259 OpenOffice, 468 OpenStack, 373 operating systems (generally), 507 operator precedence, 43 , 57, 575 optimization Cython/NumPy/C extensions for, 444 Numba for, 445 of code, 439 -446 PyPI for, 444 timing, 439 -443 OrderedDict() function, 211 ORM (Object-Relational Mapper), 327 -330 os module, 268 , 281 os.path.join() function, 274
P package managers, 412 packages, 202 -207 (see also specific packages) installing, 410 -413 installing from source, 413 module search path, 204 modules versus objects, 206 namespace packages, 205 practice exercise answers, 551
practice exercises, 216
relative/absolute imports, 205
third-party for science and math, 490 -498
packed sequences, 489
padded binary files, 317
padded text files, 303
padding characters, removing, 74
Panda3D, 458
pandas, 314
Pandas, 498
parameters
default values for, 146
defined, 143
parent class, 174
parent method, 177
path separators, 273
pathlib module, 274
pathnames, 268 , 272-275
building, 274
getting, 273
getting symlink pathnames, 273
pathlib module, 274
patterns, in searches (see regular expressions)
pdb (Python debugger), 430 -436
PDF, 468
pep8, 420
per cent sign (%), 78 -80
Perl, 13
permissions, 270
persistence (term), 303
persistent storage, 303 -342
binary files, 317
flat text files, 303
full-text databases, 341
nonvolatile, 503
NoSQL data stores, 330 -341
padded text files, 303
practice exercise answers, 560 -564
practice exercises, 342
relational databases, 318 -330
tabular text files, 304 -316
pex files, 449
pickle module, 361
Pillow, 454 -457
pip, 411
conda and, 519
installation, 517
pipenv, 412 , 518
plotting, 461 -465, 461-463
Index | 591
PNG files, 240 -242 poetry, 518 polymorphism, 188 pop() function, 103 , 124 positional arguments, 145 , 147-149 PostgreSQL, 323 pound sign (#), 53 pprint() function, 213 precedence, 43 , 57, 575 pretty printer, 213 primary key, 318 print() function debugging and, 429 quotes with, 65 write() versus, 261 writing text file with, 260 processes concurrency and, 286 creating with multiprocessing module, 279 creating with subprocess module, 278 killing, 280 practice exercise answers, 560 practice exercises, 301 process information, 282 system information, 281 programs (generally), 5 (see also processes) elements of, 5 example of complex program, 7-11 example of simple program, 5-7 proper subsets, 135 proper superset, 135 properties computed values returned by, 184 for attribute privacy, 182 -183 getting/setting attribute values, 181 -186 psutil package, 282 publish-subscribe (pub-sub) pattern, 355 -359 defined, 350 Redis, 355 -357 ZeroMQ, 357 pull pattern, 350 Puppet, 369 push pattern, 350 PyCharm, 413 pyflakes, 419 -420 pylint, 419 -420 PyPI, 444 PySimpleGUI, 460
Python (generally)
basics, 3-21
interactive interpreter and, 18
language style, 20
noninteractive programs, 19
other languages compared to, 12 -14
practice exercise answers, 527
practice exercises, 21
Python 2 versus Python 3, 17
real-world applications, 11
reasons to use, 14 -16
running, 18 -20
situations not to use, 16
Zen of, 20
Python 2, integer limits in, 49
Python 3
business applications, 467 -483
dataclasses, 196
installation, 511 -519
integer size in, 49
Linux installation, 517
macOS installation, 515 -516
math/statistics applications, 485 -500
resources for learning, 449 -452
Unicode text strings, 221 -223
Unix installation, 517
Windows installation, 516
Python community, 451
Python debugger (pdb), 430 -436
Python Image Library (PIL), 454 -457
Python Package Index (PyPI), 410
Python standard library, 207 -215
benefits of, 207
counting items, 209
deque, 211
finding code, 410
handling missing keys, 207 -209
iterating over code structures, 212
order by key, 211
pretty printer, 213
third-party alternatives to, 215
Q
Qt, 460
queues, 285
quotes, 64 -66
R
RAM (Random Access Memory), 303 , 503
592 | Index
randint() function, 214 random numbers, 214 random() function, 493 randrange() function, 215 range() function, 91 , 214 rational arithmetic, 489 raw string, 64 , 67 read() function, 262 , 265 reader() function, 305 readline() function, 263 readlines() function, 264 realpath() function, 273 record, defined, 303 recursion, 164 Redis, 332 -340 caches/expiration, 339 concurrency and, 297 -300 hashes, 336 lists, 335 publish-subscribe system with, 355 -357 sets, 336 sorted sets, 338 strings, 332 regressions, 421 regular expressions, 230 -238 all matches, 232 basics, 230 exact matches, 231 first match, 232 pattern specifiers, 235 -238 replace at match, 233 special characters, 233 -235 specifying match output, 238 split at matches, 233 relational databases, 318 -330 DB-API, 320 MySQL, 323 PostgreSQL, 323 SQL, 319 -320 SQLAlchemy, 323 -330 SQLite, 321 -323 relative access times, 504 relative imports, 205 relative/absolute imports, 205 Remote Procedure Calls (RPCs), 363 -369 gRPC, 368 JSON-RPC, 365 MessagePack RPC, 366 Twirp, 369
XML RPC, 364
Zerorpc, 367
remove() function
delete an item from list by value, 103
deleting file with, 270
deleting value from set, 131
rename() function, 269
replace() function, 73
Representational State Transfer (REST), 400
request-reply patterns, 350 -355
request-reply technique, 300
requests module, 382
requests-html library, 403
reshape() function, 493
right justification, 77
rmdir() function, 271
rows, 318
Ruby, 13
Ruby on Rails, 13
Rust, 14
S
s (step), 433
safe_load () function, 314
Salt, 369
sample() function, 214
scapy library, 349
scientific applications, 485 -500
domain-specific libraries, 498
NumPy, 490 -497
practice exercise answers, 573
practice exercises, 500
SciKit, 497
SciPy, 497
SciKit library, 497
SciPy library, 497
scraping, 401 -403
scrapy, 402
Seaborn, 464
search() function, 232
secondary index, 319
security issues
business data, 474
debug in production web servers, 393
pickle module, 362
SQL injection, 322
XML and, 308
seek() function, 265 -267, 303
self argument, 180
Index | 593
sentinel value, 298 sequences, 91 (see also lists; text strings) converting into dictionaries, 119 creating with generators, 157 deque and, 211 escape sequences, 66 extracting subsequences from lists, 99 generating number sequences, 91 iterating over multiple, 110 packed, 489 strings as, 63 tuples, 93 -97 serialization (see data serialization) servers, 509 set intersection operator (&), 133 -135 set() function, 130 setdefault() function, 207 -209 setlocale() function, 255 -257 sets, 129 -136 adding items to, 131 checking for combinations of set values, 133 -135 comprehensions, 136 conversion to, 130 creating, 130 deleting items from, 131 dictionaries versus, 129 getting length of, 131 immutable, 136 in Redis, 336 iterating over all items in, 132 null/empty, 129 practice exercise answers, 541 -544 practice exercises, 138 sorted, 338 testing for values, 132 setter methods, 181 setUp() function, 422 shapefile about, 476 -478 drawing a map from, 475 shell program, 12 slash (/), 273 slice assign values to sublist with, 102 copying lists with, 107 extracting subsequence from lists with, 99 reversing a string with, 212
substring extraction with, 70 -72
social media sites, APIs and, 361
sockets, 345 -349
software (generally)
assembler, 506
basics, 505 -510
bits/bytes, 505
cloud computing, 509
containers, 508
distributed computing/networks, 509
higher-level languages, 507
Kubernetes, 510
machine language, 505
operating systems, 507
virtual machines, 508
sort() function, 105
sorted() function, 105
source control systems, 446 -449
Git, 446 -449
Mercurial, 446
Spark, 370
special (magic) methods, 190 -193
special characters, 233 -235
split() function, 72 , 98, 233
spreadsheets, 317 , 469
SQL (structured query language), 319 -320
SQLAlchemy, 323 -330
engine layer, 325
Expression Language, 326
installing, 324
Object-Relational Mapper, 327 -330
SQLite, 321 -323
SSD (Solid State Drive), 503
stack (LIFO queue), 103
static languages, 13
static methods, 187
statistics applications, 485 -490, 497
statistics module, 489
storage (see persistent storage)
str() function, 66
strftime() function, 253
string methods, 576
string module, 578
StringIO, 275 -276
strings, 63 -85
case, 76
combining, 68 , 73
converting to list, 105
creating with quotes, 64 -66
594 | Index
creating with str(), 66 defined, 6 determining length of, 72 duplicating, 68 escape sequences and, 66 formatting, 77 -83 getting single character from, 69 in Redis, 332 practice exercise answers, 531 -535, 552-557 practice exercises, 84 , 245 regular expressions, 230 -238 searching and selecting, 75 -76 splitting, 72 string methods, 576 stripping padding characters from, 74 substring extraction with slice, 70 -72 substring substitution, 73 Unicode, 220 -229 strip() function, 74 strong typing, 25 strptime() function, 255 struct module, 240 -242 struct_time objects, 252 sub() function, 233 subdirectories, creating, 271 sublists, 102 subprocess module, 278 subsets, 134 substrings extraction with slice, 70 -72 substitution with replace(), 73 subtraction, 40 sum() function, 154 super() function, 177 superset, 135 symbolic link, 269 symlink() function, 269 symmetric_difference () function, 134 synchronous (term), 284 system() function, 281
T tables, 318 tablib, 314 tabular text files, 304 -316 configuration files, 315 CSV, 304 -306 HTML, 309 JSON, 309 -312
pandas, 314
tablib, 314
XML, 306 -309
YAML, 312 -314
TCP (Transmission Control Protocol), 344 ,
347 -349
TCP/IP (Transmission Control Protocol/Inter‐
net Protocol), 343 -350, 376
Netcat and, 349
scapy library, 349
sockets, 345 -349
tearDown() function, 422
tell() function, 265
telnet, 377
terminate() function, 280
testing, 418 -428
continuous integration, 428
doctest, 425
nose, 426
Python code checkers, 419 -420
unittest, 421 -425
text files
configuration files, 315
CSV, 304 -306
flat, 303
HTML, 309
JSON, 309 -312
pandas, 314
tablib, 314
tabular, 304 -316
XML, 306 -309
YAML, 312 -314
text strings (see strings)
threads, 287 -289, 292-295
3-D graphics, 458
3-D animation, 458
throttling, 300
TileDB, 318
time (see calendars/clocks)
time module, 251 -253
time series databases, 340
time zones, UTC versus, 253
time-to-live, 332 -340
timedelta object, 249
timeit() function, 441
timing, measurement of, 439 -443
today() function, 249
true values, 58
tuple() conversion function, 95
Index | 595
tuples, 93 -97 combining, 95 comparing, 96 concatenating, 96 creating with commas and (), 94 creating with tuple(), 95 duplicating with *, 96 generator comprehension and, 115 iteration, 96 lists versus, 114 named tuples, 195 practice exercise answers, 537 -540 practice exercises, 115 unpacking, 95 , 113 Twirp, 369 twisted framework, 295 2-D graphics, 453 -458, 453, 454-457, 457 type hints (type annotations), 418 type() function, 31 types basic, 25 booleans, 37 -51 converting to integers, 47 -48 integers, 38 -49 practice exercise answers, 528 practice exercises, 35 variables, names, and objects, 29 -33
U UDP (User Datagram Protocol), 344 underscore (_), 39 , 163 Unicode text strings, 64 , 220-229 decoding, 226 encode() function, 224 HTML entities, 227 in Python 3 strings, 221 -223 normalization, 228 practice exercise answers, 552 -557 practice exercises, 245 UTF-8 encoding, 223 unicodedata module, 221 union() function, 134 unittest, 421 -425 Unix file operations, 267 -270 path separators, 273 Python 3 installation, 517 time representation, 251 update() function, 123
UTC, 253
V
validation, data, 472
values
arguments versus parameters, 142 -151
assigning to multiple variable names, 33
assigning to sublist, 102
assigning to variable, 28
computed, 184
copying from one dictionary to another, 125
counting occurrences in list, 104
default parameter, 146
deleting from set, 131
dictionaries and, 117
extracting from list, 98
false, 58 , 144
getting/setting for attributes, 181 -186
in set, 132
key-value stores (see NoSQL data stores)
literal, 26
None, 144
returned by property, 184
sentinel, 298
test for existence in list, 104
true, 58
variables, 26
values() function, 122
variables, 26
(see also attributes)
assigning one list to more than one variable,
106
assigning value to multiple names, 33
assignment, 28
integers and, 41 -43
names versus places, 29 -33
naming, 34
reassigning names, 33
type hints (type annotations), 418
version control (see source control systems)
vertical bar (ǀ), 134
virtual machines, 508
virtualenv, 412 , 517
visual art (see graphics)
visualization, 461 -465
Bokeh, 465
Seaborn, 464
VPython, 458
596 | Index
W walrus operator, 60 web browser, 377 web clients, 376 -384 Pythons standard web libraries, 380 -382 requests module, 382 testing with curl, 378 testing with httpbin, 380 testing with httpie, 379 testing with telnet, 377 web crawling, 401 -403 web scraping, 401 -403 Web Server Gateway Interface (WSGI), 384 web servers, 384 -397 Apache, 386 ASGI, 386 automation and, 398 -400 bare-bones Python HTTP server, 384 Bottle framework, 389 -392 database frameworks, 397 -398 Django framework, 397 event-based, 388 Flask framework, 392 -396 frameworks, 388 -397 NGINX, 388 WSGI, 384 webbrowser module, 398 webview module, 399 while loops, 87 -89 canceling, 88 check break use with else, 89 skipping ahead, 88 white space, in Python program structure, 53 whitespace characters, stripping, 74 wildcard patterns (*), 230 Windows package managers, 413 path separators, 273 Python 3 installation, 516
shell, 12
World Wide Web, 375 -407
(see also web entries)
APIs and REST, 400
crawling and scraping, 401 -403
history of, 375
practice exercise answers, 571
practice exercises, 407
video search program example, 403 -406
web clients, 376 -384
web servers, 384 -397
web services and automation, 398 -400
web services/APIs, 360
write() function
print() versus, 261
writing binary file with, 264
writing text file with, 261
writeheader() function, 306
writer() function, 305
WxPython, 460
X
XML (Extensible Markup Language) format,
306 -309
xmlrpc, 364
Y
YAML (YAML Aint Markup Language),
312 -314
Z
Zen of Python, 20
ZeroMQ library, 350 -354, 357
Zerorpc, 367
zeros() function, 492
zip files, 449
zip() function, 110
zsets (sorted sets), 338
Index | 597
About the Author
Bill Lubanovic has been busy with Unix since 1977, GUIs since 1981, databases since
1990, and the web since 1993:
- 1982–1988 (Intran): Developed MetaForm on the first commercial graphic workstation.
- 1990–1995 (Northwest Airlines): Wrote a graphic yield management system; got the airline on the internet; and wrote its first internet marketing test.
- 1994 (Tela): Cofounded an early ISP.
- 1995–1999 (WAM!NET): Developed web dashboards and 3M Digital Media Repository.
- 1999–2005 (Mad Scheme): Cofounded a web development/hosting company.
- 2005 (O’Reilly): Wrote parts of Linux Server Security.
- 2007 (O’Reilly): Coauthored Linux System Administration.
- 2010–2013 (Keep): Designed and built Core Services between web frontend and database backends.
- 2014 (O’Reilly): Wrote the first edition of Introducing Python.
- 2015–2016 (Internet Archive): Worked on APIs and a Python version of the Wayback Machine.
- 2016–2018 (CrowdStrike): Managed Python-based services processing billions of daily security events.
Bill enjoys life in the Sangre de Sasquatch mountains of Minnesota with his wonder‐
ful family: wife Mary; son Tom (and wife Roxie); daughter Karin (and husband Erik);
and cats Inga, Chester, and Lucy.
Colophon
The animal on the cover of Introducing Python is a python—that is, a member of the
Python genus, which consists of 10 recognized snake species. Pythons are nonveno‐
mous, constricting snakes native to the tropics and subtropics of the Eastern
Hemisphere.
Pythons vary in length, based on species and sex, from approximately 3 feet long (ball
python) to reported cases of more than 20 feet long (reticulated python). They are
recognizable by their flat triangular-shaped heads and long backward-curving teeth.
Generally, they have a combination of brown, tan, and black skin, arranged in dia‐
mond or interlocking blotch patterns. Albino pythons are white and yellow, and
while at a disadvantage in the wild, are popular in zoos and as pets.
Pythons are strong swimmers, but these snakes almost exclusively ambush prey on
land or in trees, attacking with their fangs and then immediately wrapping them‐
selves around the quarry to asphyxiate it. Unlike boas, another well-known group of
constrictor snakes, pythons lay eggs rather than birth live young. The female python
will brood over the eggs until they hatch, shivering with her large muscular coils to
keep the eggs warm.
The Burmese python has become an invasive species in Florida, competing with the
American alligator for prey and significantly reducing the number of native birds and
small mammals in the Everglades. Along with increased popularity as a pet in the
1990s, it is thought that Hurricane Andrew in 1992 is responsible for allowing captive
Burmese pythons into the wild by destroying a zoo and breeding facility.
Most species of python have a conservation status of Least Concern, but the Bermese
(native population) and Borneo short-tailed python are currently listed as Vulnerable.
Many of the animals on O’Reilly covers are endangered; all of them are important to
the world.
The cover illustration is by Jose Marzan, based on a black and white engraving from
Johnson’s Natural History. The cover fonts are Gilroy Semibold and Guardian Sans.
The text font is Adobe Minion Pro; the heading font is Adobe Myriad Condensed;
and the code font is Dalton Maag’s Ubuntu Mono.
There’s much more
where this came from.
Experience books, videos, live online
training courses, and more from O’Reilly
and our 200+ partners—all in one place.
Learn more at oreilly.com/online-learning
©2019 O’Reilly Media, Inc. O’Reilly is a registered trademark of O’Reilly Media, Inc. | 175