Juan M. Alonso

115 días para desarmar la bomba

Thu, 28 Dec 2023 00:00:00 GMT

En un poco más de 200 páginas, Jorge Remes Lenicov, Ministro de Economía durante la Presidencia de Duhalde en la crisis post 2001, explica la hecatombe argentina de aquel entonces, cuenta los entretelones políticos detrás de su plan y define una serie de medidas para que no caigamos nuevamente en la perdición.

En este blog, intento rescatar lo más interesante de la Parte I para entender las cuestiones difíciles de comprender, al menos para los que éramos muy chicos, de esta etapa que marcó una época.

Algunas citas a continuación están incluídas textuales del libro, otras fueron parafraseadas, y algunas otras digeridas por mí.

C3: "El que despositó dólares, recibirá dólares": la estrategia económica 💵
C4: Se empieza a desactivar la bomba y la convertibilidad 💱
C5: Una decisión difícil: la pesificación ⚖️
C6: El desarme del 'corralito' 🚪
C7: La crisis de los contratos 📃
C8: Desdolarización y desindexación de las tarifas de los servicios públicos 🧾
C9: Una de las grandes batallas: equilibrio fiscal consolidado 🤸
C10: Anclaje monetario y política financiera 🏦
C11: La hora de las retenciones ⏰️
C12: La negociación con el FMI 🤝
C13: El trabajo dio sus frutos 🍋
Conclusión

C3: "El que despositó dólares, recibirá dólares": la estrategia económica 💵

El primer capítulo es un repaso de la entrada de Jorge Remes Lenicov y equipo al mando del Ministerio de Economía, y el segundo un resumen del panorama económico colapsado de aquel entonces. En el tercer capítulo la trama se pone más interesante, por esto salteo los primeros 2.

El 2 de enero de 2002, Duhalde dio su discurso inagural como Presidente ante el Congreso. En él, pronunció la frase que titula al capítulo y que quedaría en la historia. Semejante imprudencia significó un baldazo de agua fría para el autor del libro, que estaba tranquilo porque había revisado el discurso junto con Duhalde durante la noche del 31 de diciembre.

El equipo de JRL, que ya se había comprometido a asumir con un plan que repasaremos en los próximos capítulos, nunca pensó que se podrían devolver los dólares al vencimiento de las obligaciones originales, ni en un plazo razonable: eso era imposible, porque los dólares depositados por los ahorristas (o pesos convertidos) no estaban en los bancos. Habían sido prestados a familias y empresas que se encontraban con enormes dificultades para devolverlos (!).

Evidentemente, alguien convenció a Duhalde entre el 31 y el 2 de cambiar el discurso y agregar esa frase. Probablemente, se debió al temor que había ante una revuelta popular. La sociedad no tenía paciencia ni motivación para escuchar explicaciones. Lo cierto es que después JRL tuvo que desmentirlo tres días después.

Otra imprudencia

Menos de un mes después, un grupo de legisladores del Partido Justicialista y de la Unión Cívica Radical convencieron a Duhalde y a Alfonsín de enjuiciar a toda la Corte Suprema de Justicia. Más allá del razonamiento que se tuviera sobre el desempeño de sus integrantes durante el gobierno de Carlos Menem, la consecuencia no deseada de esta decisión fue el serio enfrentamiento con el máximo tribunal de justicia, que tenía en sus manos la potestad de invalidar cualquier medida del plan de emergencia.

Habano

Tal fue el error de Duhalde que JRL fue personalmente a ver al presidente de la Corte, Julio Nazareno, para explicarle que era fundamental para la economía que no se abriera el corralito porque eso podría hacer que los pesos salieran al mercado, se fueran al dólar e hicieran disparar la cotización. Nazareno, fumando un habano, le dijo: "Ahora nosotros estamos preocupados por este tema del juicio político, así que no tenemos tiempo para atender y estudiar ese pedido". Así fue como autorizaron a abrir el corralito, lloviendo cautelares y habilitando la salida de pesos del mercado.

Dejame tranquilo con mis temas

Las decisiones que JRL debía tomar requerían, según él, máxima celeridad. No podía sentarse a debatir con otros ministros que no son del área económica cada vez que correspondía tomarse una medida en cuanto a impuestos, retenciones, tipo de cambio, etc. Las discusiones que tuvo JRL a nivel interno fueron tan difíciles como las que se suscitaron con los que estaban afuera del gobierno. Duhalde lo apoyó en esto: "No presentés más el presupuesto en la reunión de gabinete. Lo arreglamos nosotros dos, después vos un día lo presentas ya cocinado, yo te cambio de tema y chau".

60 días

En sesenta días se adoptaron las diez medidas centrales. Esto sucedió gracias a:

acuerdo con los radicales para tener mayoría en el Congreso
Ley de Emergencia con muchas de la medidas
acuerdo con las provincias

Las medidas:

Medida	Implementación
Política cambiaria	Se formalizó la salida de la convertibilidad. Devaluación, flotación administrada y control de cambios parcial.
Pesificación y contratos	Conversión a pesos de las obligaciones contraídas en dólares. Las deudas se pesificaron a 1 por 1 y se actualizaron por Coeficiente de Estabilización de Referencia (CER) más tasa de interés; los depósitos se pesificaron a 1,40 por 1 y se actualizaron por CER más tasa de interés, y los contratos privados se pesificaron según costos compartidos. Ese tipo de actualización rigió solo para las deudas anteriores, pero no para las que se contrajeran en el futuro. No hubo seguro de cambios.
Desarme del «corralito»	Reprogramación de los depósitos y alternativas para retirar los fondos. Opciones para nuevos depósitos. Contraproducente actitud de la Corte Suprema de Justicia.
Política fiscal consolidada	Se rehízo el Presupuesto Nacional con superávit primario y se impulsaron reformas en la política y administración tributarias. Se firmó un acuerdo entre la Nación y las provincias para reducir el déficit provincial, iniciar el ordenamiento fiscal, normalizar el régimen de coparticipación y eliminar las cuasimonedas.
Política monetaria y financiera	Reforma de la Carta Orgánica para que el Banco Central de la República Argentina sea prestamista de última instancia y actúe en el mercado cambiario. Expansión monetaria restrictiva e inicio de operaciones de mercado abierto. Política de redescuentos para evitar la caída de bancos.
Derechos de exportación	Se establecieron de manera decreciente según el mayor el grado de elaboración.
Deuda pública	Reducción en dólares de la deuda pública interna por su pesificación.
Tarifas	Conversión a pesos, congelamiento transitorio y renegociación limitada de las concesiones.
Salarios	Se pospuso su incremento para concentrarse en la creación de empleo.
Normalización de las relaciones financieras con el mundo	Se iniciaron conversaciones con el FMI y otros organismos financieros internacionales.

C4: Se empieza a desactivar la bomba y la convertibilidad 💱

Durante la década de pizza con champagne (¿o ahora se dice 'espumante'?) la economía entera se acomodó a la paridad artificial de 1 peso, 1 dólar estadounidense. Las economías por naturaleza son dinámicas, entonces para sostener el 1 a 1 de la economía argentina se tendría que haber obedecido al comportamiento de la economía EEUU: igual o menor inflación y mayor o igual tasa de crecimiento de la productividad. Por supuesto, nada de esto ocurrió, con lo cual nuestro tipo de cambio quedó con el tiempo desactualizado deliberadamente, o "atrasado".

C5: Una decisión difícil: la pesificación ⚖️

Probablemente es este el capítulo más debatible de todos, digno de horas de café (torrado no, por favor). Como comentaba más arriba, nuestra economía estaba lejísimo de siquiera parecerse a la de EEUU. Además de que el tipo de cambio no reflejaba esto, el común de la gente no lograba enlazar la crisis que se vivía con la falsa premisa del 1 a 1.

Hago una pausa acá sobre la comprensión sobre la economía de 'la gente'. Estudié la mayor parte de mi vida y me gradué de ingeniero en informática, y aún así me llevó una enorme cantidad de horas en internet comprender conceptos básicos de finanzas y economía, incluso al leer este libro. No quiero imaginar lo que debe ser para un trabajador corriente frenar y pensar de qué se habla cuando se escucha decir "tipo de cambio", "tasa de interés", "LELIQ", etc, si es que siquiera tiene el tiempo y la energía para hacerlo. El Estado no debería regalar computadores, subsidiar internet, ni promocionar cursos de programación, sino antes garantizar que su población entienda los conceptos básicos de economía y finanzas. Ahí si seremos más poderosos y menos proclives en el futuro a caer en apuestas de teóricos que juegan con nuestros impuestos y, por transitividad, con nuestra vida.

La 'pesificación' de Roosevelt

En la década de 1930, el entonces presidente de EEUU Franklin D. Roosevelt, hizo salir al país de un tipo de cambio fijo, el patrón oro, hacia una moneda nacional, el dólar. De esto se inspiró Remes Lenicov, que remarca que FDR fue aún más duro que su pesificación: el que no depositaba su oro en la Reserva Federal iba preso. Después devaluó 70% y salió del patrón oro. La onza de oro troy pasó de 20,6 USD a 35 USD.

Como era de esperar, llovieron juicios por doquier. Pero SCOTUS le dió la razón al gobierno: nadie podía enriquecerse sin hacer nada. Entonces, si yo tenía 1000 USD antes de la salida del patrón oro, no podía tener 1700 USD al otro día. En el mismo sentido, nadie podía deber más de lo que debía por la misma cosa (los bancos había enjuiciado a los deudores).

La reingeniería

Para salir había que necesariamente afectar a todo lo denominado en dólares: depósitos, contratos, préstamos. JLV explica que esto fue de lo más difícil de la gestión, no sólo a nivel técnico sino más bien a nivel social. Claro, pesificar las obligaciones de la economía destrabó el proceso de producción, pero del otro lado del mostrador pulverizó a quienes pretendían atesorar sus ahorros en el greenback.

Casualmente, mientras escribo esto en 2023, se debate cómo dolarizar la economía argentina. Haber leído este libro y preguntarme lo mismo que JRL con el panorama actual no hace más que dejarme claro que es algo completamente inviable y caótico.

¿Qué hubiera pasado si se hubieran mantenido los depósitos y préstamos en dólares, o se hubieran transformado en pesos al tipo de cambio libre?

Una pregunta contrafáctica que se hace JRL y que responde claramente. En ese escenario, dado que la inflación de todo ese período fue menor que la devaluación, los depositantes hubieran tenido una gran ganancia de capital en pesos, mientas que los deudores se hubieran insolventado masivamente, los bancos se habrían quedado con propiedas como colateral de esas deudas, transfiriendo así la riqueza de los endeudados a los acreedores.

El valor de las cuotas de los créditos se hubieran triplicado, haciendo imposible a los deudores pagar sus deudas, y por consecuencia, haciendo imposible que los bancos devuelvan los depósitos que tomaron para prestarle a los primeros, en la moneda que sea. Al quebrar los bancos, hubieran quebrado los ahorristas, las familias no habrían pagado nada de sus hipotecas y las empresas hubieran quebrado masivamente al no poder pagar frente a tal incremento de su endeudamiento.

En diciembre de 2001, 7.5 millones de personas y empresas debían USD 38.430 millones. Coincido con JRL en que dejar esto en manos del mercado a resolverse 'mano a mano' era sencillamente inviable. Sin embargo, sí hubo arreglos ad hoc para contratos entre privados, que habían sufrido su ruptura por la crisis de la convertabilidad.

Es así que se decidió por pesificar todo, y ajustar por CER para evitar licuaciones de créditos y depósitos.

CER: en el contexto de la economía argentina, el "Coeficiente de Estabilización de Referencia" es un factor utilizado para ajustar ciertos instrumentos financieros y contratos en Argentina para tener en cuenta la inflación.

JRL explica que esto era lo más equitativo entre ahorristas y deudores, además de comenzar a producir y de realentar los depósitos en un contexto de crisis. Romper el 1 a 1 devaluación sin pesificación no hubiera sido coherente, al igual que no tener moneda propia (según quién lo diga, ¿no cierto, León?).

Si a diciembre de 2001, una persona debía 100 dólares/pesos, en febrero de 2002, ¿cuánto debía? ¿50, 150, 300? No, debía $100 más un índice de precios más una tasa de interés, es decir, unos $110 pesos. JRL así explica que no hubo licuación ni se benefició a nadie. Si se hubiera mantenido el dólar, ¡la deuda se triplicaba! Con eso quebraban todas las empresas (su patrimonio neto pasa a ser negativo). ¿Cómo se sale de una crisis con todas las empresas quebradas? Lo que se hizo fue evitar el aumento del valor real de las deudas.

A fines de 2004 el dólar estaba en $2,97, y un crédito actualizado por CER más tasa de interés exigía multiplicar po 1,98 el valor original. Evidentemente, no hubo licuación ni imposibilidad de pago.

La asimetría

Para los ahorristas, se pesificó a $1,40 por dólar por la particular razón de la promesa de Duhalde de pagarles en dólares a los depositantes. Entonces, JRL y su equipo se atuvo a la cotización del momento de la promesa. Así es que medida en pesos, los ahorristas no perdieron poder adquisitivo, pero en dólares, claramente sí. Incluso, los deudores debían menos dólares.

Es obvio que en la diferencia entre 1,40 para ahorristas y 1 para endeudados hay una pérdida que alguien paga. Por esa diferencia, se le dieron en el 2002 a los bancos bonos que costaron USD 11.000 millones. A cambio, se pesificaron las deudas internas en dólares de los bancos y de las AFJP a 1,40 más CER, de tal manera que el total bajó de USD 53.450 millones a USD 22.050. Un deal.

Hospital de empresas

El FMI, ese ente que vive en nuestra memoria colectiva permanentemente por nuesta nula capacidad de control de gasto, no fue siempre el mismo. De hecho, en esa época, tuvo un comportamiento que años después informes propios internos de la agencia fuertemente criticaron. A Remes Lenicov le propusieron un hospital de empresas, que consistía en crear un fondo estatal para absorber la deuda de las empresas, capitalizar los bancos y garantizar los depósitos. Este experimento ya había sido implementado en México, en la crisis del Tequila, con un desastre como resultado para el Estado (USD 65.000 millones), y con una fiesta de sobornos por todas partes. Imagínese lo que hubiera sido este hospital en nuestra tierra, donde hacemos un acto para inaugurar una canilla.

Other interests

También hubo presiones de otros bancos para no pesificar. En particular, los bancos de EEUU y EU, que eran mayoristas y no estaban expuestos a depositantes minoristas, preferían seguir todo como está, sabiendo que los otros bancos en la cuerda floja quebrarían, dándoles automáticamente mayor mercado.

Otras propuestas que llegaban: privatizar el Banco Nación y el Banco Provincia, y echar 500000 empleados públicos. Para la primera, JRL alude al hecho de que son bancos que "nacieron con el país", y para la segunda, que con los cacerolazos que había, era imposible.

Es fácil con el diario del lunes (y con la victoria de Milei con el 55%) criticar a JRL por ambos puntos. Pero lo hago igual, ya que quedan en evidencia las prioridades de su equipo en esos momentos determinantes: proteger a dos bancos porque "nacieron con el país" es un argumento nostálgico y débil. Hubiera esperado algo en torno a "qué buenos balances tienen" o "es ilegal por X razón". En cuanto a los empleados públicos, es aún menos convincente. En momentos donde los que menos tienen peor sufrían, se decidió proteger a empleados del Estado que cobraban de todos. JRL habla sobre "discutirla y esperar el momento", que no es coherente con sus medidas (acordes) de shock.

A título personal, me pregunto, ¿cuándo llega ese momento? ¿Cómo nos damos cuenta que estamos en ese momento de discutir tal o cual tema? En 2023, llegó sin más de la mano de la 'motosierra' de Milei, quien repartío en un primer momento el ajuste entre la población y 'la casta', cuando se creía que sería sólo sobre la última. Mejor eso que nada, mejor shock que "discutir y esperar" mientras todos los argentinos son parte de la distribución de las pérdidas, salvo los protegidos que presten servicios en el Estado.

Deudas de empresas en el extranjero

Por si faltaba poco, se sumó la presión de las empresas que tenían deudas en el exterior por casi USD 78.000 millones para que les dieran un seguro de cambio.

Seguro de cambio: también conocido como "forward contract" o "contrato a plazo de divisas", es un acuerdo financiero entre dos partes para intercambiar una cantidad específica de una moneda por otra en una fecha futura predeterminada y a un precio acordado. Este tipo de instrumento se utiliza comúnmente para gestionar el riesgo de fluctuaciones en los tipos de cambio.

Sucedía que durante los años noventa, muchas de las empresas grandes, extranjeras y nacionales, tomaron deuda en el exterior a tasas de interés mucho más bajas que en el mercado local. Esto fue un gran riesgo pero a su vez una gran ventaja frente a las empresas locales, en su mayoría pymes, que tomaban al 25% anual. Cuando 'llegó la cuenta', fueron de rodillas al Estado a pedirle un salvataje. Por la naturaleza propia del riesgo asumido, y por el costo fiscal imposible de afrontar, fue rechazado. "Te vamos a 'pegar' igual, pero te entendemos." le confesaban los empresarios a JRL.

Sistema financiero

Pondría a la protección del sistema financiero en el podio de las medidas más claves (sin dejar de ser polémica) de aquellos momentos. Con créditos incobrables y ahorristas licuados, no hay sistema que resista, y por consecuencia, país. Todo esfuerzo compartido (pesificación, devaluación) fue útil para que Argentina se normalizara después. Recomiendo fuertemente el documental PANIC! sobre cómo el rescate (siempre criticable) de los bancos de Wall Street durante la crisis de 2008 evitó un descalabro muchísimo mayor. Inside Job es la otra campana.

Con el tiempo, los ahorristas recuperaron en promedio 85% de los depósitos en dólares, al mismo tiempo que medido en pesos mejoró su poder adquisitivo. JRL señala que no hubo confiscación de depósitos, pues se ofrecieron distintas posibilidades de retiros, compras de bienes registrables, bonos en dólares, entre otras medidas, que confieso suenan más a eufemismos para ese -15% que sufrieron los ahorristas, pero que hicieron al esfuerzo compartido para salir de la convertibilidad.

C6: El desarme del 'corralito' 🚪

La dirigencia política estaba tan asustada con el "que se vayan todos" que muchos legisladores entraban al Congreso vestidos de sport para pasar desapercibidos, dentro se vestían de traje para las sesiones, y al salir nuevamente en sport y así fugar rápidamente a bordo de un taxi.

El corralito se diseñó para detener la acelerada fuga de capitales y evitar el colapso del sistema financiero. Hubo una corrida contra el peso, reduciendo las reservas del BCRA, y contra los depósitos, deteriorando a los bancos. Una combinación explosiva, tanto como la transformación automática de salarios en pesos a dólares, creando dólares sin ningún respaldo.

El 3 de diciembre de 2001, el entonces Ministro de Economía Cavallo lo estableció. Por 60 días se congelaron los depósitos, se limitaron los retiros y se incentivó la dolarización de depósitos y deudas. Para eludir controles, la gente abría cuentas a nombre de parientes o amigos. En el interín, fracasó un caje de deuda interna (fase 1 del corralito, a una tasa extrafalaria de 16%), se cumplieron los 60 días, llegó febrero de 2002 con Duhalde y Remes Lenicov y así nunca llegó a finalizar el corralito armoniosamente.

Muy por el contrario, desmultiplicar los dólares no fue sencillo. Por definición, los depósitos viven en el cortísimo plazo, mientras que el período de recuperación de los préstamos es a plazo mucho mayores. Esta diferencia temporal es insalvable en una corrida (pregúntenle sino a Silicon Valley Bank). ¿Cómo se devolvían los dólares entonces? ¿Cómo se reestablecía la cadena de pagos? No estaban en el BCRA, ni tampoco los pesos. Emitir habría generado mayor inflación.

Mientras los deudores pedía pesificación 1 a 1, los ahorristas demandaban sus depósitos de inmediato.

El 3 de diciembre de 2001 sólo quedaban 10 (sí, diez) depositantes con más de USD 500.000. El resto se había avidado antes e ido.

Se procedió a la reprogramación de los depósitos a plazo fijo, a 12 y a 24 meses, y a la liberación progresiva de las cuentas a la vista (acceso inmediato). A fin de enero ya se habían liberado la mayoría de los depósitos en pesos, y la mitad en dólares. Con "Certificados Representativos de Depósitos Reprogramados" se podían comprar casas, máquinas, vehículos. Más adentrado 2002, se ofrecieron bonos a 10 años en USD, que casi todos los ahorristas aceptaron y cobraron al final de su madurez sin inconvenientes, pero, claro, con una fe ciega en el Estado argentino por 10 veranos seguidos. Qué agallas, a ojos de hoy, después de tantos defaults y reestructuraciones.

Se excluye el desembolso del FMI de septiembre de 2001 en las reservas. El aumento en los depósitos de enero de 2002 se debe a la pesificación.

La Corte Suprema: "pido"

Mientras avanzaba la salida del corralito paulatinamente, el máximo tribunal sentenció sin apreciar la delicada situación económica que el corralito debía abrirse. No era esta vez un lobbysta cualquiera, sino la primera autoridad del Poder Judicial. Pero la Corte fue más allá cuando aceptó que jueces otorgaran cautelares para retirar la totalidad de los fondos sin permitir la defensa de los bancos en cuestión. Varios jueces terminaron enjuiciados.

Esto produjo filtraciones en los depósitos, distorsionando innecesariamente la cotización del dólar entorpeciendo el acceso al crédito de familias y empresas.

Sin los cambios realizados en el corralito y con la mecánica de las cauterales, solo habrían cobrado los primeros en llegar, y no hubiera quedado ni un peso para el resto.

Bancos offshore

Algunos proponían, además de dolarizar, que la banca argentina sea offshore, que dependiera de otro país y que mantuviera los dólares. En este punto JRL argumenta lo mismo que Sergio Massa en el debate 2023: no hay país que no maneje su sistema financiero, como tampoco hay quienes no tengan su propia moneda, salvo algunos muy pequeños. No me convence del todo como justificativo, pero si el resto del mundo tuvo crisis como las nuestras y salió a flote con moneda propia, ¿por qué deberían ir con esta estrategia offshore? Pienso en Alemania post-guerra con hiperinflación, Brasil en los 90s, Japón últimamente calibrando sus cuentas con el yen, etc. Todas situaciones muy complejas pero que no requirieron saltar a una dolarización.

Coyuntura

JRL cierra el capítulo diciendo que cuando hay una crisis muy profunda de conyuntura, como la de 2001, hasta que no se normalice la situación no se pueden tomar medidas estructurales, detallando que el acuerdo entre Duhalde y Alfonsín fue para salir de la convertibilidad y crecer sin inflación, nada más (ni nada menos).

Las primeras semanas del gobierno del Presidente Javier Milei y del Ministro Luis Caputo a fines de 2023 fueron un reto a esta idea. En el momento que escribo estas líneas, queda por verse si el súper DNU que se dictaminó para liberar la economía es aprobado y hecho efectivo, y si sus medidas alcanzan para normalizar la inflación y pobreza rampantes. Sólo el destino lo sabrá.

C7: La crisis de los contratos 📃

Desde fines del 2000 se fue extendiendo una profunda crisis de contratos entre particulares, sumados a las obligaciones con el Estado que se dejaron de pagar (los impuestos). El Estado fue dejando de brindar prestaciones y de pagar sus deudas. Los bancos no cobraban créditos ni tampoco devolvían depósitos.

Todo ello fue consecuencia de la generalizada iliquidez, la quiebra del Estado, la ruptura de la cadena de pagos y la aparición de las cuasimonedas. Una macroeconomía absolutamente detonada por los incumplimientos.

El economista sueco Axel Leijonhufvud dice: “Cuando los reclamos son masivos […] y los reclamos exceden lo que es posible pagar, tienen que tomarse decisiones sobre la distribución de pérdidas.” Sin esta distribución la economía no vuelve a funcionar. Lo que es materialmente imposible no es exigible.

Desde aquí nació en el equipo de JRL la Ley de Emergencia, por la cual nadie debía beneficiarse de la crisis, y todos debían soportar los costos equitativamente. El haber sincerado la relación peso-dólar permitió restablecer, muy progresivamente, la cadena de pagos y la recaudación fiscal.

Años después, la Corte Suprema no sólo convalidó la constitucionalidad de la ley y sus medidas, sino además aclaró que no haberlas hecho hubiera sido peor. El derecho es siempre una pretensión de encauzar las conductas sociales en cierta dirección. Cuando esa dirección se desvía, el derecho debe modificarse. No se puede insistir en un deber ser contrario a la dirección de la realidad.

Esto cobra más relevancia que nunca a finales de 2023, cuando el DNU de demás de 600 puntos deba ser tratado por el Congreso. Eventualmente, la Corte Suprema puede llegar a expedirse sobre el decreto. Hace más de 20 años apoyó el paquete de medidas para salir de la crisis. ¿Y esta vez?

C8: Desdolarización y desindexación de las tarifas de los servicios públicos 🧾

Hubo un momento donde los llamados hacia el Ministerio de Economía de JRL llegaban del otro lado del Atlántico. Resulta que muchas de las empresas que habían entrado durante las privatizaciones de los 90s era de origen español. Felipe González, ex presidente español, hablaba con los radicales. El rey Juan Carlos lo llamaba a Duhalde. Rodrigo Ratto, vicepresidente segundo de España, hablaba con JRL.

Su intención en un primer momento, durante la efímera presidencia de Rodriguez Saá, era postergar todos los cambios al próximo año, de tal manera que no afectasen sus balances para el año corriente. Luego pasaron a presionar para continuar con tarifas atadas al dólar, para continuar con la convertibilidad, y hasta para dolarizar directamente la economía. Curiosas exigencias, puesto que estas empresas tuvieron en los años noventa más del doble de rentabilidad que el resto de las grandes empresas. Además, en el peor período, 1999-2001, mientras el resto sufrió pérdidas, estas continuaban con altas ganancias, todo debido a la absurda indexación de las tarifas a la inflación de EEUU.

JRL y su equipo permanecieron convencidos que las tarifas iban a ser pesificadas y congeladas transitoriamente. De hecho, de no hacerlo, las tarifas se hubieran triplicado, y en esa realidad no quedaría ni un solo usuario para pagar su tarifa. En otras palabras, la pesificación terminó salvando a las compañías en cuestión. Más adelante, se fueron ajustando las tarifas según el costo de la devaluación. Imposible que todos queden conformes, pero la situación se pudo superar. No por tanto.

Hablá con Julio

Un bendito día de 2003, el tema de las tarifas quedó a cargo de Julio De Vido, en la presidencia de Nestor Kirchner. Con todo el intelecto que los caracterizaba, decidieron subsidiar la oferta, es decir, a las empresas. Fue una decisión pésima, no sólo por el atraso que produjo que subsistió décadas, sino porque está estudiado que los subsidios deben siempre ir a la demanda. Al subsidiar a la oferta, todos lo usuarios, ricos y pobres, pagan menos por sus tarifas, cuando lo correcto sería que sólo aquellos que verdaderamente lo necesiten lo reciban. Demás está mencionar las sospechas de corrupción entorno a estas maniobras a favor de la oferta.

“Decile a Jorge que afloje”

Hay que pensar que había en juego miles de millones de dólares. Para abril de 2023, los subsidios eran del 4% del PBI, lo que significa unos u$s 20.000 millones. No es broma. Por ese dinero se puede llegar a hacer cualquier cosa. Por eso, no extrañó el despliegue que se hizo por parte de las empresas para tratar de doblarles el brazo.

Pero también había comentarios de funcionarios del gobierno y legisladores a los que acudían amigos comunes: «Decile a Jorge que afloje», ese era el mensaje.

JRL debe ser recordado cuanto menos por esta actitud, por no rendirse ante los que pedían “que afloje”. Interpretaría el “que afloje” más cercano a “que no toque mi parte”.

C9: Una de las grandes batallas: equilibrio fiscal consolidado 🤸

Comenzó el año 2002 con un déficit financiero del 5,7% del PBI. Era insostenible a nivel Nación y a nivel provincial. ¿Financiamiento internacional? Bien, gracias. Esta situación acarreaba una disminución de la recaudación del 8,9% para el año 2001. El famoso Diciembre de ese año se tocó un piso: una baja del 28%.

Diez provincias, sin fondos suficientes, emitieron cuasimonedas al notar que el BCRA no podía emitir más del stock de reservas. Las transferencias de Nación que les correspondían a las provincias fueron suplantadas por un bonito mecanismo financiero: las LECOP, bonos que circulaban como medio de pago, pero no se podían usar para depósitos y créditos.

Quiero default. ¿De qué sabor?

El mundo funciona aún con déficits fiscales. Muchas veces, estos son mayores al PBI. Los mercados internacionales los financian, primordialmente porque confían en la capacidad de repago de esas naciones (EEUU, Japón, EU). No fue el caso por estas latitudes. Teníamos default implícito generado por el canje de títulos públicos de mayo de 2001, por el canje forzoso de la deuda interna en octubre o porque el FMI dejó de hacer desembolsos en noviembre de ese mismo año. También teníamos default explícito declarado a fines de 2001 por Rodriguez Saá.

Fue imperioso entonces diferenciar la deuda interna de la externa para reconstruir el sistema de pagos. Con el arreglo y reducción en dólares de la deuda interna (USD $22.050 millones), se compensó el costo de la asimetría de la pesificación entre depósitos y créditos. Me pregunto, entonces, a cuánto se hubiera pesificado de no haber ajustado esta deuda: ¿1,80? ¿2,00?

Así fue que en marzo se pesificaron y pospusieron por tres años (!) todas las obligaciones en dólares que estaban bajo legislación argentina utilizando el mismo criterio de los depósitos: $1,40 más CER más tasas de interés. Claro, si hubiera sido bajo legislación extranjera (Nueva York, por ejemplo) el juicio más pequeño era del tamaño del Taj Mahal.

Algo no menor es el impacto de la devaluación. Sumado al impacto positivo en las exportaciones, se previó un aumento en los ingresos fiscales, producto del aumento en el nivel de precios post-devaluación. Esto también contribuyó a la reducción del déficit.

Bajar el déficit de 5% a 3% es una tarea algo rápida, mientras que bajarlo de 3% a 0% es una obra monumental. Con todas estas medidas (Pacto Nación-Provincias) y reprogramaciones (no gastar más de lo que se colecta, es decir, erogar menos de los que ingresa) finalizó 2002 con un superávit primario de 0,7% y un déficit financiero de menos de la mitad del recibido: 1,5%. Las provincias hicieron un esfuerzo mayor al pactado, de manera que sólo 2 no llegaron a la metas acordadas. Underpromise, overdeliver.

La dékada ganada

El sector público argentino inició así un período de ordenamiento y equilibrio fiscal como no se había registrado en las últimas décadas. Fue en esta etapa que se erigieron los superávits gemelos de los que tanto se vanaglorió después Néstor Kirchner, que en definitiva había heredado de la gestión de JRL. Es más: JRL y equipo tuvieron alto superávit comercial cuando todavía el precio de la soja no se había disparado, porque eso ocurrió recién a partir de 2003.

Había que ser austeros y no prometer lo que no se podía cumplir.

C10: Anclaje monetario y política financiera 🏦

La crisis del sistema financiero era de una gravedad sin precedentes. Llegando al 2002, los créditos al sector privado, en su mayoría en dólares, habían caído 20%. Las tasas reales de interés eran del 25% anual. La incobrabilidad llegaba al 25%, y como sucede siempre en estos casos, tanto a principio de siglo como en 2023, los bancos preferían prestarle al sector público.

Durante la gran crisis de los años treinta, desapareció el 30% de los bancos de EEUU.

Para estabilizar los precios y ayudar en la recuperación económica, no bastaba con la política fiscal del capítulo anterior. La política monetaria y financiera fue igualmente importante. Los objetivos eran:

Impedir que la explosión de la convertibilidad terminara en hiperinflación
Circunscribir el overshooting (o sobrereacción) cambiario
Evitar la caída de bancos
Ayudar a definir una tasa de interés de referencia
Plantear pasos a seguir para recuperar el sistema financiero

La idea era simple: controlar la cantidad de pesos que ingresaban al mercado para evitar que fueran a la compra de dólares con un equilibrio tal que esa maniobra no entorpeciera la reactivación. Es así que se les dieron redescuentos a bancos para otorgar créditos baratos y así expandir la demanda interna. Algunos decían que el plan no tenía ancla monetaria, mientras que otros decían que el ancla era ortodoxa y recesiva. Otros sostenía que no había que dar redescuentos, pues el mercado acomodaría todo.

En situaciones como estas, hay opiniones e interpretaciones infinitas. Según como uno corte los datos, puede argumentar una mirada o la otra. Esto sucede en la actualidad también, con lecturas opuestas pero simultáneas en cuanto al soft landing que planea la Reserva Federal de EEUU en su política monetaria post-pandemia. Según a quien leas, se viene el fin del mundo y hay que vender todo, o se debe buy the dip porque estamos en bull market y "el resto no la ve".

Visita del diputado Carbonetto

El diputado del Polo Social Daniel Carbonetto visitó a pedido de Duhalde a JRL, para decirle en el BCRA había dinero suficiente para aumentar salarios y el gasto público sin necesidad de emitir, y que oponerse a esto era entonces algo sin fundamento. Luego de perdile algunas aclaraciones, a JRL le cayó la ficha: Carbonetto confundió emisión con impresión. Efectivamente, había dinero en el BCRA, pero eran los encajes mínimos de los bancos (!).

Nacen las LEBAC

En la ejecución del programa, se absorbió el desvío de pesos con venta de reservas y, a partir de marzo de 2002, a través de la creación de Letras del BCRA (LEBAC), que se usaron para definir la tasa de interés de referencia. Veinte años después estas letras (y otras con nombres estrambóticos) siguen girando.

Pese a estas dificultades, hubo una expansión monetaria neta y del circulante muy moderada, y no se terminó convalidando la inflación en el tiempo. No menor fue el hecho de que no se cayó ningún banco y, en los años siguientes, fueron devolviendo todos los redescuentos.

Cuasimonedas: "jefe, me siento mal..."

Las cuasimonedas, creadas por la Nación y once provincias, funcionaban como medios de pago pero no podían ser usadas para depósitos y créditos. Por esta razón, su impacto sobre la circulación monetaria y sobre el efecto multiplicador era limitado. No se comportan como la emisión de monedas tradicionales, por eso son cuasimonedas.

JRL recuerda una anécdota de un gobernador que le explicó al Ministro por qué no era necesario pensar cómo eliminar su cuasimoneda: "No te preocupes, Jorge, van a ir desapareciendo solas. Los bonos fueron impresos en papel común, entonces se van a ir desgastando con el uso y van a desaparecer". Para reir y llorar.

C11: La hora de las retenciones ⏰️

Por si faltaban presiones que tironeen, había llegado la hora de las retenciones a las exportaciones. Con la devaluación a $1,40, no había espacio para retenciones. Pero cuando el dólar llegó a $3, se impusieron con una pauta decreciente según el valor agregado de los productos. El objetivo fue múltiple y terminaron conviviendo consideraciones de corto y largo plazo:

aumentar ingresos fiscales y financiar programas sociales (45% de las retenciones)
favorecer industrialización del país
limitar transferencias de ingresos
atemperar el efecto de la devaluación sobre los precios internos de los alimentos

A pesar de las retenciones, todas las actividades fueron recuperando la rentabilidad. Con respecto a lo último, los exportadores no se vieron perjudicados porque el tipo de cambio real efectivo multilateral fue del doble del vigente durante la convertibilidad. Esto me sorprendió, porque, como muchos, creía que las retenciones siempre perjudicaban a los exportadores. Pero no es así, porque eso depende más del tipo de cambio efectivo. En la década de los noventa, no había prácticamente retenciones pero como el tipo de cambio estaba atrasado, en 2001 la mayoría de los productores habían quebrado o tenían tierras hipotecadas. Por el contrario, a partir de 2002, con las retenciones, pero con un tipo de cambio competitivo, la producción agropecuaria creció aceleradamente.

De todas maneras, había que tener cuidado con el nivel de las alícuotas. Un estudió reveló que retenciones más altas habían promovido la subfacturación y la evasión impositiva. Estas discusiones no vinieron sin presiones, inclusive, en el propio gabinete: el entonces secretario de Agricultura terminó renunciando. También algunos gobernadores de provincias agrícolas y petroleras primero comenzando pidiendo que no se pusieran las retenciones, para luego terminar pidiendo que se coparticipara en su recaudación, aunque por precepto constitucional las rentas de aduana son recursos nacionales no coparticipables. Por su parte y como de costumbre, la mayoría de los legisladores no opinaba públicamente para no quemarse.

Al tiempo de estas medidas, dirigentes rurales habían propuesto donar el 5% del valor exportado a programas sociales a cambio de 0 retenciones. JRL se plantó: "Nosotros vamos por el 20% y no hay donación que valga, porque esto es una decisión autónoma del Estado."

En simultáneo, la Sociedad Rural Argentina se preguntaba, y con razón, por qué no se reducía el gasto público pero sí se les imponían las retenciones. En esto JRL no ahondó. Me parece que faltó un párrafo aparte para argumentar.

C12: La negociación con el FMI 🤝

A la hora de negociar con el FMI, había dos problemas: el primero era que no creían lo que se les decía desde la Argentina porque las administraciones anteriores les habían mentido; y el segundo era que le reprochaban al "Mecon" que ya habían implementado un programa económico sin consultarles nada ni darles participación alguna, algo no menor si uno repasa las negociaciones de Macri, Guzmán, Massa y Caputo.

Fuera como fuere, la negociación con el FMI era clave para posteriormente destrabar arreglos con el Banco Mundial, el Banco Interamericano de Desarrollo y otros organismos internacionales, que entre 2002 y 2003 ascendían a USD 25.699 millones.

El FMI apuntó naturalmente contra la pesificación asimétrica puesto que hacía menos probable el cumplimiento de los pagos (versus dolarizar o liberar el dólar), y propuso como alternativa el curioso hospital de empresas del capítulo 5. Terminaron modificando su posición, e incluso, en marzo, cambiaron a los funcionarios que tenían a su cargo el tema de Argentina. Pedían que se avanzara más rápido y hasta querían saber las fechas precisas de aprobación e implementación de las próximas medidas y leyes (sic), aumentando las exigencias antes de iniciar cualquier negociación.

Después de idas y vueltas, el Fondo pedía por:

liberación del tipo de cambio y libre movilidad de capitales sin intervención ni controles
no desdolarizar no congelar las tarifas
abrir rápidamente el "corralito"
crear el "hospital de empresas"
no dar redescuentos a los bancos
ajuste fiscal con superávit primario durante un año, entre otras

Para JRL y su equipo, el FMI simplemente quería dolarizar y no ayudar al país. Por esto, no le quedaban dudas de que debían seguir sólos (sin esperar nuevos desembolsos). No hubo acuerdo. La Argentina fue el único caso en el mundo que no sólo recibió fondos frescos, sino que debió pagar al conjunto de los organismos internacionales. A todos los países que habían estado en dificultades unos años antes (México, Indonesia, Turquía) les dieron más de USD 10.000 millones a cada uno.

El Fondo tuvo en esa época una actitud dogmática, confusa e incluso caótica, algo que fue reconocido en 2004 por la Oficina de Evaluación Independiente del propio organismo, que puso en manifiesto los serios errores y la actitud adversa y negativa del FMI durante la crisis.

JRL incluye otras propuestas del FMI que me llaman la atención luego de la victoria de La Libertad Avanza en 2023:

arancelamiento universitario
despedir a 500.000 empleados públicos
modificar el régimen de las obras sociales
reforma impositiva, laboral y de la Coparticipación Federal

Evidentemente, en 2002 el ajuste no tocaba demasiado el timbre del Estado.

C13: El trabajo dio sus frutos 🍋

Crisis de la Convertibilidad

El resultado verdaderamente fue satisfactorio. Si se repasa la historia, entre 1960 y 2001 se registaron veintinueve años con déficit fiscal y en cuenta corriente, y en los 13 años restantes hubo un superávit solo en cuenta corriente. En 2002 se revirtió el resultado del déficit de cuenta corriente, y en los años siguientes, por primera vez en 50 años, se registró superávit en ambos sectores.

El esquema de la pesificación asimétrica permitió, poco a poco, ir recomponiendo el tejido social y productivo. Los que más ganaron en términos de poder adquisitivo en pesos fueron los que tenían dóalres en el exterior o en el "colchón". Los que más perdieron, como de costumbre en todas las grandes crisis, fueron quienes tenían ingresos fijos: trabajadores y jubilados.

¿Había forma de evitar injusticias post-covertibilidad?

No creo que existían salidas indoloras. Sigo creyendo que no las hay aún hoy para salir de la crisis durante 2023. El autor solo puede decir que en esa coyuntura y con el poco tiempo disponible no había medidas diferentes o mejores que las que tomaron, ni mecanismos más adecuados para atemperar costos. Se dio a los secotres más necesitados la máxima protección posible con los pocos instrumentos disponibles.

¿Cómo es que no hubo una hiperinflación?

Gran preocupación del Presidente banfileño Duhalde. Los resguardos estaban para evitarla. La devaluación fue muy importante pero no se aumentaron los salarios ni las tarifas y se desindexó la economía; también se llevó adelante una política monetaria muy prudente, el resultado fiscal primario fue positivo y se redujo a menos de la mitad el déficit financiero, mientras que el altísimo superávit comercial permitió la recomposición de las reservas. Se hizo exactamente lo contrario de lo realizado en las anteriores crisis.

Inflación 2002-2005, microscópica comparada con la del último gobierno kirchnerista

La demanda de empleo y la actividad industrial se reactiva rápidamente. Base 2000 = 100

La salida

A Remes Lenicov le hubiera gustado quedarse luego de 115 días, pero no es la condiciones de ese entonces. Duhalde, que no debe haber querido quedar en la historia como un presidente peronista que ajustó, quiso comenzar a emitir todo lo posible para aumentar salarios, dar créditos baratos, romper con el mundo, fijar el tipo de cambio y controlar precios. JRL presenta su renuncia allí mismo. Los gobernadores, que tampoco querían cambiar la política monetaria, presionaron para continuar el rumbo con el Documentos de los 14 puntos. Duhalde luego le pidió que se quedara, pero ya era tarde: la renuncia es una renuncia.

Fue una pena para JRL y su equipo, porque después tuvieron que escuchar a Nestor Kirchner y la tropa austral decir que todo empezó cuando llegaron ellos en mayo del 2003. El entonces Jefe de Gabinete, Alberto Fernández, recibió una carta de JRL explicándole qué habían hecho en esos 4 meses para que no digan semejante barbaridad, a lo que el primero le contestó: "Sí, tenés razón, pero es lo que vamos a seguir diciendo". Ahí nació "el relato".

Conclusión

La historia te pone en en el lugar que ella quiere, y no en el que uno pretende.

Sobre la crisis, y en mi humilde opinión, hubiera procurado distribuir el costo de la salida del 2001 más sobre el Estado que sobre la población, pero reconozco que la conyuntura era difícil y que el Estado debía seguir prestando servicios, sobre todo a los más necesitados. JRL agrega que hubiera sido contraproducente y que se corría el riesgo de perder todo lo hecho para ordenar la macro. En ese momento, la sociedad no tenía paciencia ni motivación para escuchar explicaciones. Creo que ahora sí la tiene, y, es más, la necesita (comparar las explicaciones de la herencia recibida de Macri vs Milei).

En varias partes JRL sobreutiliza lo contrafáctico para justificar las acciones que lideró. Por ejemplo, cuando comenta como "descabellado porque conducía al caos" la propuesta del FMI en las negociaciones del 2002. Aún así, tampoco me parece justo recaerle en esto cuando en semejante crisis hay que tomar decisiones contundentes en el momento, y no se dispone de un largo tiempo para experimentar y analizar. Él mismo reitera que la velocidad de la aplicación de las soluciones era tan importante como la solución misma.

El libro es un relato formidable de cuatro meses vertiginosos, contado desde el punto de vista de un protagonista central de la historia. Complementa con un gran resumen de los últimos 20 años, y cierra con pautas clave hacia un desarrollo sustentable. Hayas vivido esos años o no, es un racconto primordial para entender cómo llegamos al presente.

Cuando la situación es compleja, nadie quiere poner la cara. Los aprietas diversos pero constantes son parte de las reglas del juego. Estas reglas estuvieron, están y seguirán estando vigentes en nuestro país.

Salir del 2001 no fue un trabajo sucio: se hizo lo que se pensaba que se debía hacer.

Insights in 'Designing Data-Intensive Applications' - Part 3/3

Sun, 10 Dec 2023 00:00:00 GMT

Welcome to my blog on the book "Designing Data Intensive Applications" by Martin Kleppmann. In this post, I will share with you key principles, trade-offs, and technologies that I found as the most interesting for building robust and scalable data systems.

Most of these notes are as they appear in the original book, or tailored by me, but all ideas are a creation of Martin or authors Martin quotes in his book.

This is Part 3. Check Part 1, Foundations of Data Systems, here, and Part 2, Distributed Data, here.

Derived Data

Derived Data

Intro

The first two parts went from the layout of the data on disk to the limits of distributed consistency. However, a single database was assumed at all times. In real life, access patterns are more complex, and there is no one database that can satisfy simultaneously different use cases. Thus, applications use a combination of stores, caches and mechanisms.

In this final part, we'll have a look at the complexities around integrating multiple different systems. If a vendor offers you a product that can "satisfy all your needs", think again.

Two Categories for store-and-process systems

Systems of record Also known as "source of truth", it's the place where new data comes in and is represented exactly once, normally normalized.
Derived data systems Existing data is taking from one system and processed in some way i.e. a cache, denormalized values, indexes, materialized views. Good read performance.

10. Batch-oriented dataflows

Types of Systems:

Services (online) A service waits for a request or instruction from a client to arrive. When one is received, the service tries to handle it as quickly as possible and sends a response back. Response time is usually the primary measure of performance of a service, and availability is often very important (if the client can’t reach the service, the user will probably get an error message).
Batch processing (offline) Such system takes a large amount of input data, runs a job to process it, Since normally there isn’t a user waiting for the job to finish, batch jobs are often scheduled to run periodically. The primary performance measure of a batch job is usually throughput (the time it takes to crunch through an input dataset of a certain size). MapReduce is for instance a batch processing algorithm.
Stream processing (near-real-time) This type is somewhere between the previous two. Like a batch processing system, a stream processor consumes inputs and produces outputs. However, a stream job operates on events shortly after they happen, whereas a batch job operates on a fixed set of input data. This difference allows stream processing systems to have lower latency than the equivalent batch systems.

MapReduce workflows

A batch job’s output is only considered valid when the job has completed successfully (MapReduce discards the partial output of a failed job). To handle these dependencies between job executions, various workflow schedulers for Hadoop have been developed, including Oozie, Azkaban, Airflow and Luigi.

While MPP (massively parallel processing) databases focus on parallel execution of analytic SQL queries on a cluster of machines, the combination of MapReduce and a distributed filesystem provides something much more like a general-purpose operating system that can run arbitrary programs.

Joins

When we talk about joins in the context of batch processing, we mean resolving all occurrences of some association within a dataset. For example, we assume that a job is processing the data for all users simultaneously, not merely looking

Batch join example.

Making random-access requests over the network for every record you want to process is too slow. Moreover, querying a remote database would mean that the batch job becomes nondeterministic, because the data in the remote database might change while the job is running.

Thus, a better approach would be to take a copy of the user database and to put it in the same distributed filesystem as the log of user activity events. You would then have the user database in one set of files in HDFS and the user activity records in another set of files, and you could use MapReduce to bring together all of the relevant records in the same place and process them efficiently. If the input datasets are partitioned into multiple files, each could be processed with multiple mappers in parallel.

Sessionization

A common use for grouping is collating all the activity events for a particular user session, in order to find out the sequence of actions that the user took—a process called sessionization. For example, such analysis could be used to work out whether users who were shown a new version of your website are more likely to make a purchase than those who were shown the old version (A/B testing), or to calculate whether some marketing activity is worthwhile.

Handling skew

Collecting all activity related to a celebrity (e.g., replies to something they posted) in a single reducer can lead to significant skew (also known as hot spots)—that is, one reducer that must process significantly more records than the others. Since a MapReduce job is only complete when all of its mappers and reducers have completed, any subsequent jobs must wait for the slowest reducer to complete before they can start.

If a join input has hot keys, there are a few algorithms you can use to compensate. For example, the skewed join method in Pig first runs a sampling job to determine which keys are hot.

Output of Batch Workflows

Where does batch processing fit in? It is not transaction processing, nor is it analytics. It is closer to analytics, in that a batch process typically scans over large portions of an input dataset. However, a workflow of MapReduce jobs is not the same as a SQL query used for analytic purposes. The output of a batch process is often not a report, but some other kind of structure.

Philosophy of outputs

By treating inputs as immutable and avoiding side effects (such as writing to external databases), batch jobs not only achieve good performance but also become much easier to maintain:

ease of rolling back: being able to recover from buggy code ('human fault tolerance')
feature development can proceed more quickly than in an environment where mistakes could mean irreversible damage. This principle of minimizing irreversibility is beneficial for Agile software development
automatic retry is safe because inputs are immutable and outputs from failed tasks are discarded by the MapReduce framework

Designing for frequent faults

At Google, a MapReduce task that runs for an hour has an approximately 5% risk of being terminated to make space for a higher-priority process. This rate is more than an order of magnitude higher than the rate of failures due to hardware issues, machine reboot, or other reasons. At this rate of preemptions, if a job has 100 tasks that each run for 10 minutes, there is a risk greater than 50% that at least one task will be terminated before it is finished.

And this is why MapReduce is designed to tolerate frequent unexpected task termination: it’s not because the hardware is particularly unreliable, it’s because the freedom to arbitrarily > terminate processes enables better resource utilization in a computing cluster.

Downsides

MapReduce’s approach of fully materializing (writing to disk) intermediate state has downsides compared to other batch mechanisms like Unix pipes:

A MapReduce job can only start when all tasks in the preceding jobs (that generate its inputs) have completed
Mappers are often redundant: they just read back the same file that was just written by a reducer
Storing intermediate state in a distributed filesystem means those files are replicated across several nodes, which is often overkill for such temporary data

Dataflow Engines

In order to fix these problems, several new engines were developed, like Spark. They handle an entire workflow as one job, rather than breaking it up into independent subjobs. Since they explicitly model the flow of data through several processing stages, these systems are known as dataflow engines. Like MapReduce, they work by repeatedly calling a user-defined function to process one record at a time on a single thread. They parallelize work by partitioning inputs, and they copy the output of one function over the network to become the input to another function.

Unlike in MapReduce, these operators need not take the strict roles of alternating map and reduce, but instead can be assembled in more flexible ways. One option is to repartition and sort records by key, like in the shuffle stage of MapReduce. This feature enables sort-merge joins and grouping in the same way as in MapReduce.

Advantages

Expensive work such as sorting need only be performed in places where it is actually required, rather than always happening by default between every map and reduce stage.
There are no unnecessary map tasks, since the work done by a mapper can often be incorporated into the preceding reduce operator (because a mapper does not change the partitioning of a dataset).
Because all joins and data dependencies in a workflow are explicitly declared, the scheduler has an overview of what data is required where, so it can make locality optimizations. For example, it can try to place the task that consumes some data on the same machine as the task that produces it, so that the data can be exchanged through a shared memory buffer rather than having to copy it over the network.
It is usually sufficient for intermediate state between operators to be kept in memory or written to local disk, which requires less I/O than writing it to HDFS (where it must be replicated to several machines and written to disk on each replica). MapReduce already uses this optimization for mapper output, but dataflow engines generalize the idea to all intermediate state.
Operators can start executing as soon as their input is ready; there is no need to wait for the entire preceding stage to finish before the next one starts.
Existing Java Virtual Machine (JVM) processes can be reused to run new operators, reducing startup overheads compared to MapReduce (which launches a new JVM for each task).

Fault Tolerance

Spark, Flink, and Tez avoid writing intermediate state to HDFS, so they take a different approach to tolerating faults: if a machine fails and the intermediate state on that machine is lost, it is recomputed from other data that is still available.

Nondeterminism when recomputing data needs to be removed in order to reliably recover from faults, for example by generating pseudorandom numbers using a fixed seed.

Recomputing data is not always the right answer: if the intermediate data is much smaller than the source data, or if the computation is very CPU-intensive, it is probably cheaper to materialize the intermediate data to files than to recompute it.

Thanks to the framework, your code in a batch processing job does not need to worry about implementing fault-tolerance mechanisms: the framework can guarantee that the final output of a job is the same as if no faults had occurred, even though in reality various tasks perhaps had to be retried. These reliable semantics are much stronger than what you usually have in online services that handle user requests and that write to databases as a side effect of processing a request.

11. Streams Processing: Batch but with lower delays.

A complex system that works is invariably found to have evolved from a simple system that works. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work.

John Gall, Systemantics (1975)

The problem with daily batch processes is that changes in the input are only reflected in the output a day later, which is too slow for many impatient users. To reduce the delay, we can run the processing more frequently—say, processing a second’s worth of data at the end of every second—or even continuously, abandoning the fixed time slices entirely and simply processing every event as it happens. That is the idea behind stream processing.

Crucially, in batch processing jobs, the input data is bounded: it has a known, fixed size (for example, it consists of a set of log files at some point in time, or a snapshot of a database’s contents). Because it is bounded, a job knows when it has finished reading the entire input, and so a job eventually completes when it is done. In stream processing, the input is unbounded. In this case, a job is never complete, because at any time there may still be more work coming in. Both are similar in some respects, but the assumption of unbounded streams also changes a lot about how we build systems

Some examples that are naturally represented as streams are user activity events, sensors providing periodic readings, and data feeds (e.g., market data in finance).

The writes to a database can also be thought of as a stream: we can capture the changelog—i.e., the history of all changes made to a database—either implicitly through change data capture or explicitly through event sourcing. Log compaction allows the stream to retain a full copy of the contents of a database.

Transmitting Event Streams

In a stream processing context, a record is more commonly known as an event, but it is essentially the same thing: a small, self-contained, immutable object containing the details of something that happened at some point in time.

In streaming terminology, an event is generated once by a producer (also known as a publisher or sender), and then potentially processed by multiple consumers (subscribers or recipients). In a filesystem, a filename identifies a set of related records; in a streaming system, related events are usually grouped together into a topic or stream.

Direct messaging from producers to consumers

UDP multicast is widely used in the financial industry for streams such as stock market feeds, where low latency is important.

Load-balancing vs fan-out patterns

Partitioned Logs

Sending a packet over a network or making a request to a network service is normally a transient operation that leaves no permanent trace. Message brokers are built around this transient messaging mindset. Databases and filesystems take the opposite approach: everything that is written to a database or file is normally expected to be permanently recorded.

A key feature of batch processes, is that you can run them repeatedly, since the input is read-only. This is not the case with message brokers: receiving a message is destructive if the acknowledgment causes it to be deleted from the broker, so you cannot run the same consumer again and expect to get the same result.

Further, if you add a new consumer to a messaging system, it typically only starts receiving messages sent after the time it was registered. We need a hybrid solution.

Using logs for message storing

Combining the durable storage approach of databases with the low-latency notification facilities of messaging, we get the hybrid behind log-based message brokers. Message brokers and event logs serve as the streaming equivalent of a filesystem.

Apache Kafka, Amazon Kinesis Streams, and Twitter’s DistributedLog are log-based message brokers that work like this. Even though these message brokers write all messages to disk, they are able to achieve throughput of millions of messages per second by partitioning across multiple machines, and fault tolerance by replicating messages.

Topic partitioning

Since partitioned logs typically preserve message ordering only within a single partition, all messages that need to be ordered consistently need to be routed to the same partition. For example, an application may require that the events relating to one particular user appear in a fixed order. This can be achieved by choosing the partition for an event based on the user ID of that event (in other words, making the user ID the partitioning key).

You can keep derived data systems such as search indexes, caches, and analytics systems continually up to date by consuming the log of changes and applying them to the derived system.

When consumers cannot keep up with producers

Even if a consumer does fall too far behind and starts missing messages, only that consumer is affected; it does not disrupt the service for other consumers. This fact is a big operational advantage: you can experimentally consume a production log for development, testing, or debugging purposes, without having to worry much about disrupting production services. When a consumer is shut down or crashes, it stops consuming resources—the only thing that remains is its consumer offset.

Search on streams

Conventional search engines first index the documents and then run queries over the index. By contrast, searching a stream turns the processing on its head: the queries are stored, and the documents run past the queries. In the simplest case, you can test every document against every query, although this can get slow if you have a large number of queries. To optimize the process, it is possible to index the queries as well as the documents, and thus narrow down the set of queries that may match.

Event time versus processing time

Confusing event time and processing time leads to bad data. For example, say you have a stream processor that measures the rate of requests (counting the number of requests per second). If you redeploy the stream processor, it may be shut down for a minute and process the backlog of events when it comes back up. If you measure the rate based on the processing time, it will look as if there was a sudden anomalous spike of requests while processing the backlog, when in fact the real rate of requests was steady.

Windowing

Whose clock are you using, anyway?

The timestamp on the events should really be the time at which the user interaction occurred, according to the mobile device’s local clock. However, the clock on a user-controlled device often cannot be trusted, as it may be accidentally or deliberately set to the wrong time. The time at which the event was received by the server (according to the server’s clock) is more likely to be accurate, since the server is under your control, but less meaningful in terms of describing the user interaction.

To adjust for incorrect device clocks, one approach is to log three timestamps:

The time at which the event occurred, according to the device clock
The time at which the event was sent to the server, according to the device clock
The time at which the event was received by the server, according to the server clock

By subtracting the second timestamp from the third, you can estimate the offset between the device clock and the server clock (assuming the network delay is negligible). You can then apply that offset to the event timestamp, and thus estimate the true time at which the event actually occurred.

Session window

Unlike other window types, a session window has no fixed duration. Instead, it is defined by grouping together all events for the same user that occur closely together in time, and the window ends when the user has been inactive for some time (for example, if there have been no events for 30 minutes). Sessionization is a common requirement for website analytics.

Stream enrichment

It is natural to think of the user activity events as a stream, and to perform a join with a user-profile database on a continuous basis within a stream processor: the input is a stream of activity events containing a user ID, and the output is a stream of activity events in which the user ID has been augmented with profile information about the user.

To perform this join, we can the have a local copy of the database in the stream processor be an in-memory hash table if it is small enough, or an index on the local disk. With this approach, it can be queried locally without a network round-trip.

Consider the Twitter timeline example. We don't want to iterate over all the people the user is following, find their recent tweets, and merge them to build it. Instead, we want a timeline cache: a kind of per-user “inbox” to which tweets are written as they are sent, so that reading the timeline is a single lookup.

Sloooowly Changing Dimension

For example, if you sell things, you need to apply the right tax rate to invoices, which depends on the country or state, the type of product, and the date of sale (since tax rates change from time to time). When joining sales to a table of tax rates, you probably want to join with the tax rate at the time of the sale, which may be different from the current tax rate if you are reprocessing historical data.

If the ordering of events across streams is undetermined, the join becomes nondeterministic, which means you cannot rerun the same job on the same input and necessarily get the same result: the events on the input streams may be interleaved in a different way when you run the job again.

In data warehouses, this issue is known as a slowly changing dimension (SCD), and it is often addressed by using a unique identifier for a particular version of the joined record: for example, every time the tax rate changes, it is given a new identifier, and the invoice includes the identifier for the tax rate at the time of sale. This change makes the join deterministic, but has the consequence that log compaction is not possible, since all versions of the records in the table need to be retained.

Fault Tolerance

The batch approach to fault tolerance ensures that, although restarting tasks means that records may in fact be processed multiple times, the visible effect in the output is as if they had only been processed once. This principle is known as exactly-once semantics.

The same issue arises in stream processing, but it is less straightforward to handle: waiting until a task is finished before making its output visible is not an option, because a stream is infinite and so you can never finish processing it.

Microbatching and checkpointing

One solution is to break the stream into small blocks, and treat each block like a miniature batch process. This approach is called microbatching, and it is used in Spark Streaming. The batch size is typically around one second, which is the result of a performance compromise: smaller batches incur greater scheduling and coordination overhead, while larger batches mean a longer delay before results of the stream processor become visible.

However, as soon as output leaves the stream processor (for example, by writing to a database, sending messages to an external message broker, or sending emails), the framework is no longer able to discard the output of a failed batch. In this case, restarting a failed task causes the external side effect to happen twice, and microbatching or checkpointing alone is not sufficient to prevent this problem.

Idempotence

Our goal is to discard the partial output of any failed tasks so that they can be safely retried without taking effect twice.

An idempotent operation is one that you can perform multiple times, and it has the same effect as if you performed it only once. For example, setting a key in a key-value store to some fixed value is idempotent (writing the value again simply overwrites the value with an identical value), whereas incrementing a counter is not idempotent (performing the increment again means the value is incremented twice).

For example, when consuming messages from Kafka, every message has a persistent, monotonically increasing offset. When writing a value to an external database, you can include the offset of the message that triggered the last write with the value. Thus, you can tell whether an update has already been applied, and avoid performing the same update again.

Rebuilding state after a failure

Any stream process that requires state—for example, any windowed aggregations (such as counters, averages, and histograms) and any tables and indexes used for joins—must ensure that this state can be recovered after a failure.

In some systems, network delay may be lower than disk access latency, and network bandwidth may be comparable to disk bandwidth. There is no universally ideal trade-off for all situations, and the merits of local versus remote state may also shift as storage and networking technologies evolve.

12. Ideas for reliable, scalable and maintable applications

Schema Migration in a... railway? 🚂

In the 19th century, there were multiple competing standards for the gauge of railway tracks in England, which created limitations in the train network. To convert tracks to a single standard gauge without disrupting train services for an extended period, a solution was found in the form of dual or mixed gauge tracks, where a third rail was added. This allowed for a gradual conversion as trains of both gauges could operate on the line. Once all trains were converted to the standard gauge, the nonstandard rail could be removed. This approach, although expensive, enabled the gradual change of gauge over several years. As a result, nonstandard gauges still exist today, such as in the BART system in the San Francisco Bay Area.

Just like with an extra gague, derived views offer a method of gradually evolving a dataset without the need for an abrupt migration. Rather than immediately switching to a new schema, the old and new schemas can coexist as separate derived views of the same underlying data. By gradually shifting a small number of users to the new view while most users continue to use the old view, the performance and bugs of the new view can be tested. Over time, more users can be transitioned to the new view until eventually the old view can be dropped. This gradual approach allows for easy reversibility at each stage, reducing the risk of irreversible damage and increasing confidence in system improvements.

Lambda architecture

If batch processing is used to reprocess historical data, and stream processing is used to process recent updates, then how do you combine the two?

With the lambda approach, the stream processor consumes the events and quickly produces an approximate update to the view; the batch processor later consumes the same set of events and produces a corrected version of the derived view. The reasoning behind this design is that batch processing is simpler and thus less prone to bugs, while stream processors are thought to be less reliable and harder to make fault-tolerant. Moreover, the stream process can use fast approximate algorithms while the batch process uses slower exact algorithms.

It does have some disadvantages, like having to maintain the same logic to run both in a batch and in a stream processing framework.

Designing Applications Around Dataflow

Unbundled databases: unifying writes across different systems

Within a single database, creating a consistent index is a built-in feature. When we compose several storage systems, we similarly need to ensure that all data changes end up in all the right places, even in the face of faults. Making it easier to reliably plug together storage systems (e.g., through change data capture and event logs) is like unbundling a database’s index-maintenance features in a way that can synchronize writes across disparate technologies.

Even spreadsheets have dataflow programming capabilities that are miles ahead of most mainstream programming languages. In a spreadsheet, you can put a formula in one cell (for example, the sum of cells in another column), and whenever any input to the formula changes, the result of the formula is automatically recalculated. This is exactly what we want at a data system level: when a record in a database changes, we want any index for that record to be automatically updated, and any cached views or aggregations that depend on the record to be automatically refreshed.

Stream processors and services

Say a customer is purchasing an item that is priced in one currency but paid for in another currency. In order to perform the currency conversion, you need to know the current exchange rate. This operation could be implemented in two ways:

In the microservices approach, the code that processes the purchase queries an exchange-rate service.
In the dataflow approach, the code that processes purchases would subscribe to a stream of exchange rate updates ahead of time, and record the current rate in a local database whenever it changes. When it comes to processing the purchase, it only needs to query the local database. Not only is the dataflow approach faster, but it is also more robust to the failure of another service. Instead of RPC, we now have a stream join between purchase events and exchange rate update events.

The fastest and most reliable network request is no network request at all!

The join is time-dependent: if the purchase events are reprocessed at a later point in time, the exchange rate will have changed. If you want to reconstruct the original output, you will need to obtain the historical exchange rate at the original time of purchase. No matter whether you query a service or subscribe to a stream of exchange rate updates, you will need to handle this time dependence.

Expressing dataflows as transformations from one dataset to another also helps evolve applications: if you want to change one of the processing steps, for example to change the structure of an index or cache, you can just rerun the new transformation code on the whole input dataset in order to rederive the output. Similarly, if something goes wrong, you can fix the code and reprocess the data in order to recover.

Observing Derived State

Pushing state changes to clients

Traditional web pages rely on the user manually reloading the page to see updated data from the server, resulting in stale information. However, newer protocols such as server-sent events and WebSockets allow for real-time communication between the server and the browser, enabling the server to actively push messages and updates to the client. This reduces the staleness of the client-side state and extends the write path all the way to end-user devices. While clients still need to use a read path to initially retrieve their state, they can subsequently rely on a continuous stream of state changes from the server. This concept of actively pushing state changes can be extended beyond datacenters to end-user devices.

The devices will be offline some of the time, and unable to receive any notifications of state changes from the server during that time. But that problem is solved: with consumer offsets, a consumer of a log-based message broker can reconnect after failing or becoming disconnected, and ensure that it doesn’t miss any messages that arrived while it was disconnected. The same technique works for individual users, where each device is a small subscriber to a small stream of events.

Search Index

In order to extend the write path all the way to the end user, we would need to fundamentally rethink the way we build many of these systems: moving away from request/response interaction and toward publish/subscribe dataflow. If you are designing data systems, keep in mind the option of subscribing to changes, not just querying the current state.

Aiming for Correctness

If your application can tolerate occasionally corrupting or losing data in unpredictable ways, life is a lot simpler, and you might be able to get away with simply crossing your fingers and hoping for the best. On the other hand, if you need stronger assurances of correctness, then serializability and atomic commit are established approaches, but they come at a cost: they typically only work in a single datacenter (ruling out geographically distributed architectures), and they limit the scale and fault-tolerance properties you can achieve.

The End-to-End Argument for Databases

Just because an application uses a data system that provides comparatively strong safety properties, such as serializable transactions, that does not mean the application is guaranteed to be free from data loss or corruption. For example, if an application has a bug that causes it to write incorrect data, or delete data from a database, serializable transactions aren’t going to save you.

Exactly-once execution of an operation

Processing twice is a form of data corruption: it is undesirable to charge a customer twice for the same service (billing them too much) or increment a counter twice (overstating some metric). In this context, exactly-once means arranging the computation such that the final effect is the same as if no faults had occurred, even if the operation actually was retried due to some fault.

One of the most effective approaches to this is to make the operation idempotent. However, taking an operation that is not naturally idempotent and making it idempotent requires some effort and care: you may need to maintain some additional metadata (such as the set of operation IDs that have updated a value), and ensure fencing when failing over from one node to another.

Uniquely identifying requests

To make the request idempotent through several hops of network communication, it is not sufficient to rely just on a transaction mechanism provided by a database—you need to consider the end-to-end flow of the request.

For example, you could generate a unique identifier for a request (such as a UUID) and include it as a hidden form field in the client application, or calculate a hash of all the relevant form fields to derive the request ID. If the web browser submits the POST request twice, the two requests will have the same request ID. You can then pass that request ID all the way through to the database and check that you only ever execute one request with a given ID. If a transaction attempts to insert an ID that already exists, an INSERT would fail and the transaction would be aborted, preventing it from taking effect twice.

Transaction Abstraction

Transactions have long been seen as a good abstraction. They take a wide range of possible issues (concurrent writes, constraint violations, crashes, network interruptions, disk failures) and collapse them down to two possible outcomes: commit or abort.

When we refuse to use distributed transactions because they are too expensive, we end up having to reimplement fault-tolerance mechanisms in application code. Reasoning about concurrency and partial failure is difficult and counterintuitive, and so if you attempt this in your applications, it most probably won't work correctly. The consequence is lost or corrupted data.

For these reasons, Martin is right in thinking it is worth exploring fault-tolerance abstractions that make it easy to provide application-specific end-to-end correctness properties, but also maintain good performance and good operational characteristics in a large-scale distributed environment.

Uniqueness constraints require consensus

If there are several concurrent requests with the same value, the system somehow needs to decide which one of the conflicting operations is accepted, and reject the others as violations of the constraint. The most common way of achieving this consensus is to make a single node the leader, and put it in charge of making all the decisions.

Uniqueness checking can be scaled out by partitioning based on the value that needs to be unique. For example, if you need to ensure uniqueness by request ID, you can ensure all requests with the same request ID are routed to the same partition.

However, asynchronous multi-master replication is ruled out, because it could happen that different masters concurrently accept conflicting writes, and thus the values are no longer unique. If you want to be able to immediately reject any writes that would violate the constraint, synchronous coordination is unavoidable

Multi-partition request processing

Ensuring that an operation is executed atomically, while satisfying constraints, becomes more interesting when several partitions are involved. Imagine a payments scenario where there are potentially three partitions: the one containing the request ID, the one containing the payee account, and the one containing the payer account. There is no reason why those three things should be in the same partition, since they are all independent from each other.

It turns out that equivalent correctness can be achieved with partitioned logs, and without an atomic commit:

The request to transfer money from account A to account B is given a unique request ID by the client, and appended to a log partition based on the request ID.
A stream processor reads the log of requests. For each request message it emits two messages to output streams: a debit instruction to the payer account A (partitioned by A), and a credit instruction to the payee account B (partitioned by B). The original request ID is included in those emitted messages.
Further processors consume the streams of credit and debit instructions, deduplicate by request ID, and apply the changes to the account balances.

Steps 1 and 2 are necessary because if the client directly sent the credit and debit instructions, it would require an atomic commit across those two partitions to ensure that either both or neither happen. To avoid the need for a distributed transaction, we first durably log the request as a single message, and then derive the credit and debit instructions from that first message. The request either appears in the log or it doesn’t, without any need for a multi-partition atomic commit.

Timeliness and Integrity

The term consistency conflates two different requirements that are worth considering separately:

Timeliness Timeliness means ensuring that users observe the system in an up-to-date state. We saw previously that if a user reads from a stale copy of the data, they may observe it in an inconsistent state. However, that inconsistency is temporary, and will eventually be resolved simply by waiting and trying again.

The CAP theorem uses consistency in the sense of linearizability (a writer waits until a transaction is committed, and thereafter its writes are immediately visible to all readers), which is a strong way of achieving timeliness. Weaker timeliness properties like read-after-write consistency can also be useful.

Integrity Integrity means absence of corruption; i.e., no data loss, and no contradictory or false data. In particular, if some derived dataset is maintained as a view onto some underlying data, the derivation must be correct. For example, a database index must correctly reflect the contents of the database—an index in which some records are missing is not very useful.

On your credit card statement, it is not surprising if a transaction that you made within the last 24 hours does not yet appear—it is normal that these systems have a certain lag. We know that banks reconcile and settle transactions asynchronously, and timeliness is not very important here. However, it would be very bad if the statement balance was not equal to the sum of the transactions plus the previous statement balance (an error in the sums), or if a transaction was charged to you but not paid to the merchant (disappearing money). Such problems would be violations of the integrity of the system.

Loosely interpreted constraints

As discussed previously, enforcing a uniqueness constraint requires consensus, typically implemented by funneling all events in a particular partition through a single node. This limitation is unavoidable if we want the traditional form of uniqueness constraint, and stream processing cannot avoid it.

However, another thing to realize is that many real applications can actually get away with much weaker notions of uniqueness:

Many airlines overbook airplanes in the expectation that some passengers will miss their flight, and many hotels overbook rooms, expecting that some guests will cancel. In these cases, the constraint of “one person per seat” is deliberately violated for business reasons, and compensation processes (refunds, upgrades, providing a complimentary room at a neighboring hotel) are put in place to handle situations in which demand exceeds supply. Even if there was no overbooking, apology and compensation processes would be needed in order to deal with flights being cancelled due to bad weather or staff on strike—recovering from such issues is just a normal part of business.

These applications do require integrity: you would not want to lose a reservation, or have money disappear due to mismatched credits and debits. But they don’t require timeliness on the enforcement of the constraint: if you have sold more items than you have in the warehouse, you can patch up the problem after the fact by apologizing.

How many apologies?

Although strict uniqueness constraints require timeliness and coordination, many applications are actually fine with loose constraints that may be temporarily violated and fixed up later, as long as integrity is preserved throughout. In other words, they reduce the number of apologies you have to make due to inconsistencies, but potentially also reduce the performance and availability of your system, and thus potentially increase the number of apologies you have to make due to outages. You cannot reduce the number of apologies to zero, but you can aim to find the best trade-off for your needs—the sweet spot where there are neither too many inconsistencies nor too many availability problems.

This approach is much more scalable and robust than the traditional approach of using distributed transactions, and fits with how many business processes work in practice.

"But hardware never fails!"

It seems unlikely, but if you have enough devices running your software, even very unlikely things do happen. Besides random memory corruption due to hardware faults or radiation, certain pathological memory access patterns can flip bits even in memory that has no faults —an effect that can be used to break security mechanisms in operating systems (this technique is known as rowhammer). Once you look closely, hardware isn’t quite the perfect abstraction that it may seem.

HDFS and Amazon S3 do not fully trust disks: they run background processes that continually read back files, compare them to other replicas, and move files from one disk to another, in order to mitigate the risk of silent corruption.

It is important to try restoring from your backups from time to time—otherwise you may only find out that your backup is broken when it is too late and you have already lost data. Don’t just blindly trust that it is all working.

Designing for Auditability

Today, weaker consistency guarantees became the norm under the banner of NoSQL, and less mature storage technologies became widely used. Yet, because the audit mechanisms had not been developed, we continued building applications on the basis of blind trust, even though this approach had now become more dangerous.

For example, for an event log, we can use hashes to check that the event storage has not been corrupted. For any derived state, we can rerun the batch and stream processors that derived it from the event log in order to check whether we get the same result, or even run a redundant derivation in parallel.

A deterministic and well-defined dataflow also makes it easier to debug and trace the execution of a system in order to determine why it did something. If something unexpected occurred, it is valuable to have the diagnostic capability to reproduce the exact circumstances that led to the unexpected event—a kind of time-travel debugging capability.

Do the Right Thing

Kleppmann is accurate in writing that a technology is not good or bad in itself—what matters is how it is used and how it affects people. I think it is not sufficient for software engineers to focus exclusively on the technology and ignore its consequences: the ethical responsibility is ours to bear also. Think about this as you get closer to users' personal data.

On a personal note, I have always looked contemptuously at recruiters pushing me towards positions for "the next big thing", which mostly ended up masquerading as an online casino app, a dodgy mental health app, or scammy crypto platforms. Chosing the companies one works for (or even applies to) is also an act that cannot escape from this responsability.

Reasoning about ethics is difficult, but it is too important to ignore.

Not-so-data-driven

A blind belief in the supremacy of data for making decisions is not only delusional, it is positively dangerous. As data-driven decision making becomes more widespread, we will need to figure out how to make algorithms accountable and transparent, how to avoid reinforcing existing biases, and how to fix them when they inevitably make mistakes.

Privacy

If the service is funded through advertising, the advertisers are the actual customers, and the users’ interests take second place. Tracking data becomes more detailed, analyses become further-reaching, and data is retained for a long time in order to build up detailed profiles of each person for marketing purposes.

As a thought experiment, try replacing the word data with surveillance, and observe if common phrases still sound so good. How about this: “In our surveillance-driven organization we collect real-time surveillance streams and store them in our surveillance warehouse. Our surveillance scientists use advanced analytics and surveillance processing in order to derive new insights.”

When sharing medical data, there are clear risks to privacy, but there are also potential opportunities: how many deaths could be prevented if data analysis was able to help us achieve better diagnostics or find better treatments? Over-regulation may prevent such breakthroughs. It is difficult to balance such potential opportunities with the risks.

The End

If you have reached this far, thank you! The purpose of these blog posts is to, through Kleppmann's magnificent book and ideas, spark different thoughts that you may have not encountered yet as a software engineer. Let's work together to build better services and products, and ultimately to leave the world a better place than we found it.

Insights in 'Designing Data-Intensive Applications' - Part 2/3

Sat, 09 Dec 2023 00:00:00 GMT

Most of these notes are as they appear in the original book, or tailored by me, but all ideas are a creation of Martin or authors Martin quotes in his book.

This is Part 2. Check Part 1, Foundations of Data Systems, here, and Part 3, Derived Data, here.

Distributed Data

Distributed Data

In the first part, we visited aspects of data systems that apply when data is stored on a single machine. For this second part, we move up a level and ask: what happens if multiple machines are involved in storage and retrieval of data?

Intro

Why should I distribute?

Scalability If your data volume, read load, or write load grows bigger than a single machine can handle, you could spread the load across multiple machines.
Fault tolerance/High Availability If your application needs to continue working even if one machine (or several machines, or the network, or an entire datacenter) goes down, you can use multiple machines to give you redundancy.
Latency If you have users around the world, you might want to have servers at various locations worldwide so that each user can be served from a datacenter that is geographically close to them (think about Netflix).

Scaling to Higher Load

If all you need is to scale to higher load, the simplest approach is to buy a more powerful machine (vertical scaling or scaling up). The problem with a shared-memory approach is that the cost grows faster than linearly.

Shared-Nothing Architectures

Also called horizontal scaling or scaling out, this approach makes each node use its CPUs, RAM, and disks independently. Any coordination done at the software level, using a conventional network.

No special hardware is required, so you can use whatever machines have the best price/performance ratio. You can potentially distribute data across multiple geographic regions, and thus reduce latency for users and potentially be able to survive the loss of an entire datacenter.

Shared-nothing architecture also implies additional complexity for applications and sometimes limits the expressiveness of the data models you can use.

5. Replication

It offers:

high availability: keeping the system running if one machines goes down
allowing an application to continue working when there is a network interruption
latency: placing data geographically close to users, for faster interaction
scalability: performing reads on replicas

Different from Partitioning

Partitioning means splitting a big database into smaller subsets called partitions so that different partitions can be assigned to different nodes (also known as sharding).

Following the leader

Every write to the database needs to be processed by every replica; otherwise, the replicas would no longer contain the same data. The most common solution for this is called leader-based replication (also known as active/passive or master–slave). Whenever the leader writes new data to its local storage, it also sends the data change to all of its followers as part of a replication log or change stream. Each follower takes the log from the leader and updates its local copy of the database accordingly, by applying all writes in the same order as they were processed on the leader.

How many leaders? Yes…

There are three main approaches:

Single-leader replication Clients send all writes to a single node (the leader), which sends a stream of data change events to the other replicas (followers). Reads can be performed on any replica, but reads from followers might be stale. No conflicts! Great for read-heavy workloads. Only makes sense with asynchronous replication, which in turn translates into temporal outdated data (eventual consistency).
Multi-leader replication Clients send each write to one of several leader nodes, any of which can accept writes. The leaders send streams of data change events to each other and to any follower nodes.
Leaderless replication Clients send each write to several nodes, and read from several nodes in parallel in order to detect and correct nodes with stale data. The client may get different responses from different nodes. Version numbers are used to determine which value is newer.

Multi-leader and leaderless replication can be more robust in the presence of faulty nodes, network interruptions, and latency spikes—at the cost of being harder to reason about (concurrency allows for conflicts) and providing only very weak consistency guarantees. If you want synchronous conflict detection, you might as well just use single-leader replication.

Sync or Async? Semi!

If a synchronous follower becomes unavailable or slow, one of the asynchronous followers is made synchronous. This guarantees that you have an up-to-date copy of the data on at least two nodes: the leader and one synchronous follower. This configuration is sometimes also called semi-synchronous.

Although asynchronous replication can be fast when the system is running smoothly, it’s important to figure out what happens when replication lag increases and servers fail. If a leader fails and you promote an asynchronously updated follower to be the new leader, recently committed data may be lost.

Durability vs Write Throughput

If the leader fails and is not recoverable, any writes that have not yet been replicated to followers are lost. This means that a write is not guaranteed to be durable, even if it has been confirmed to the client. However, a fully asynchronous configuration has the advantage that the leader can continue processing writes, even if all of its followers have fallen behind.

Weakening durability may sound like a bad trade-off, but asynchronous replication is nevertheless widely used, especially if there are many followers or if they are geographically distributed.

We lost the Leader: Failover

Handling a failure of the leader is trickier: one of the followers needs to be promoted to be the new leader, clients need to be reconfigured to send their writes to the new leader, and the other followers need to start consuming data changes from the new leader.

Discarding writes is especially dangerous if other storage systems outside of the database need to be coordinated with the database contents. For example, in one incident at GitHub, an out-of-date MySQL follower was promoted to leader. The database used an auto-incrementing counter to assign primary keys to new rows, but because the new leader’s counter lagged behind the old leader’s, it reused some primary keys that were previously assigned by the old leader. These primary keys were also used in a Redis store, so the reuse of primary keys resulted in inconsistency between MySQL and Redis, which caused some private data to be disclosed to the wrong users.

Problems with Replication

In the move to distributed (replicated and partitioned) databases, many systems have abandoned transactions, claiming that they are too expensive in terms of performance and availability, and asserting that eventual consistency is inevitable in a scalable system.

A few consistency models helpful for deciding how an application should behave under replication lag:

Read-after-write consistency: read your own writes Users should always see data that they submitted themselves. Always read the user’s own profile from the leader, and any other users’ profiles from a follower. Monitor the replication lag on followers and prevent queries on any follower that is more than one minute behind the leader.
Monotonic reads Make sure that each user always makes their reads from the same replica. For example, the replica can be chosen based on a hash of the user ID. However, if that replica fails, the user’s queries will need to be rerouted to another replica.
Consistent prefix reads Users should see the data in a state that makes causal sense: for example, seeing a question and its reply in the correct order. One solution is to make sure that any writes that are causally related to each other are written to the same partition—but in some applications that cannot be done efficiently.

happens-before relationship: whenever you have two operations A and B, there are three possibilities: either A happened before B, or B happened before A, or A and B are concurrent.

Amazin' conflict

For some time, the conflict resolution logic on the Amazon shopping cart would preserve items added to the cart, but not items removed from the cart. Thus, customers would sometimes see items reappearing in their carts even though they had previously been removed.

6. Partitioning

When you have so much data that storing and processing it on a single machine is no longer feasible, you can partition it to spread the data and query load evenly across multiple machines, avoiding hot nodes (scalability).

Approaches

Key range partitioning, where sorting keys has the advantage that efficient range queries are possible. Partitions are typically rebalanced dynamically by splitting the range into two subranges when a partition gets too big (against hot ranges)
Hash partitioning, where a hash function is applied to each key, and a partition owns a range of hashes. Range queries become inefficient, but may distribute load more evenly.
Hybrid, with a compound key: using one part of the key to identify the partition and another part for the sort order.

Consistent Hashing

It is a way of evenly distributing load across an internet-wide system of caches such as a CDN. It uses randomly chosen partition boundaries to avoid the need for central control or distributed consensus. Note that consistent here has nothing to do with replica consistency or ACID consistency, but rather describes a particular approach to rebalancing.

This particular approach actually doesn’t work very well for databases, so it is rarely used in practice. Because this is so confusing, it’s best to avoid the term consistent hashing and just call it hash partitioning instead.

Secondary indexes

A secondary index can be partitioned in two ways:

Document-partitioned (local), where the secondary indexes are stored in the same partition as the primary key and value. This means that only a single partition needs to be updated on write, but a read of the secondary index requires a scatter/gather across all partitions.
Term-partitioned (global), where the secondary indexes are partitioned separately, using the indexed values. An entry in the secondary index may include records from all partitions of the primary key. When a document is written, several partitions of the secondary index need to be updated; however, a read can be served from a single partition.

Amazon DynamoDB states that its global secondary indexes are updated (asynchronously) within a fraction of a second in normal circumstances.

Rebalancing

Data storage, read and write requests (the “load”) should be shared fairly between nodes.
While rebalancing is happening, the database should continue accepting reads and writes.
No more data than necessary should be moved between nodes.

How not to do it: hash mod N

If the number of nodes N changes, most of the keys will need to be moved from one node to another. For example, say hash(key) = 123456. If you initially have 10 nodes, that key starts out on node 6 (because 123456 mod 10 = 6). When you grow to 11 nodes, the key needs to move to node 3 (123456 mod 11 = 3), and when you grow to 12 nodes, it needs to move to node 0 (123456 mod 12 = 0). Such frequent moves make rebalancing excessively expensive.

A better way is to do dynamic partitioning. If there is only a small amount of data, a small number of partitions is sufficient, so overheads are small; if there is a huge amount of data, the size of each individual partition is limited to a configurable maximum.

Routing queries to the appropriate partition

How does the component making the routing decision (which may be one of the nodes, the routing tier, or the client) learn about changes in the assignment of partitions to nodes? This is a challenging problem, because it is important that all participants agree—otherwise requests would be sent to the wrong nodes and not handled correctly. Many distributed data systems rely on a separate coordination service such as ZooKeeper.

Using ZooKeeper to keep track of assignment of partitions to nodes.

7. Transactions

Transactions are an abstraction layer that allows an application to pretend that certain concurrency problems and certain kinds of hardware and software faults don’t exist.

Conceptually, all the reads and writes in a transaction are executed as one operation: either the entire transaction succeeds (commit) or it fails (abort, rollback). If it fails, the application can safely retry.

Not every application needs transactions, and sometimes there are advantages to weakening transactional guarantees or abandoning them entirely (for example, to achieve higher performance or higher availability).

For complex access patterns, transactions can hugely reduce the number of potential error cases you need to think about. Without transactions, various error scenarios (processes crashing, network interruptions, power outages, disk full, unexpected concurrency, etc.) mean that data can become inconsistent in various ways.

Isolation Levels and Famous Race Conditions

Read Committed

The most basic level of transaction isolation. It makes two guarantees:

When reading from the database, you will only see data that has been committed (no dirty reads).
When writing to the database, you will only overwrite data that has been committed (no dirty writes).

Dirty reads One client reads another client’s writes before they have been committed.

No dirty reads.

Dirty writes One client overwrites data that another client has written, but not yet committed.

Dirty writes.

Snapshot Isolation/Repeatable Read

There are still plenty of ways in which you can have concurrency bugs even with read committed isolation.

Read skew/Non-repeatable reads A client sees different parts of the database at different points in time. This issue is most commonly prevented with snapshot isolation, which allows a transaction to read from a consistent snapshot corresponding to data committed in one particular point in time.

Read skew.

Serializable isolation

The read committed and snapshot isolation levels are about the guarantees of what a read-only transaction can see in the presence of concurrent writes. What kind of conflicts can occur between concurrently writing transactions?

Lost updates Two clients concurrently perform a read-modify-write cycle. One overwrites the other’s write without incorporating its changes, so data is lost.

Lost updates.

Write skew A transaction reads something, makes a decision based on the value it saw, and writes the decision to the database. However, by the time the write is made, the premise of the decision is no longer true.

Write skew.

Another example around preventing double-spending: a service that allows users to spend money or points needs to check that a user doesn’t spend more than they have. You might implement this by inserting a tentative spending item into a user’s account, listing all the items in the account, and checking that the sum is positive. With write skew, it could happen that two spending items are inserted concurrently that together cause the balance to go negative, but that neither transaction notices the other.

Phantom reads A transaction reads objects that match some search condition. Another client makes a write that affects the results of that search. Snapshot isolation prevents straightforward phantom reads, but phantoms in the context of write skew require special treatment, such as index-range locks.

Serializable isolation

Usually regarded as the strongest isolation level. It guarantees that even though transactions may execute in parallel, the end result is the same as if they had executed serially, thus, preventing race conditions.

But if serializable isolation is so much better than the mess of weak isolation levels, why isn’t everyone using it?! Most databases that provide serializability today use one of three techniques.

Literally executing transactions in a serial order

Every transaction must be small and fast, because it takes only one slow transaction to stall all transaction processing.
Whenever the dataset can fit in memory.
Write throughput must be low enough to be handled on a single CPU core

Two-phase locking (2PL) Several transactions are allowed to concurrently read the same object as long as nobody is writing to it. But as soon as anyone wants to modify or delete an object, exclusive access is required. In 2PL, writers don’t just block other writers; they also block readers and vice versa: it's pessimistic.

If transaction A has read an object and transaction B wants to write to that object, B must wait until A commits or aborts before it can continue. (This ensures that B can’t change the object unexpectedly behind A’s back.)
If transaction A has written an object and transaction B wants to read that object, B must wait until A commits or aborts before it can continue.

Serializable snapshot isolation (SSI) On the one hand, we have implementations of serializability that don’t perform well (two-phase locking) or don’t scale well (serial execution). On the other hand, we have weak isolation levels that have good performance, but are prone to various race conditions (lost updates, write skew, phantoms, etc.).

A fairly new algorithm (SSI) avoids most of the downsides of the previous approaches. It uses an optimistic approach, allowing transactions to proceed without blocking. When a transaction wants to commit, it is checked, and it is aborted if the execution was not serializable.

Snapshot isolation has the mantra "readers never block writers", and "writers never block readers", which captures this key difference between snapshot isolation and two-phase locking. Because 2PL provides serializability, it protects against all the race conditions, including lost updates and write skew.

If the database keeps track of each transaction’s activity in great detail, it can be precise about which transactions need to abort, but the bookkeeping overhead can become significant. Less detailed tracking is faster, but may lead to more transactions being aborted than strictly necessary.

The rate of aborts significantly affects the overall performance of SSI. For example, a transaction that reads and writes data over a long period of time is likely to run into conflicts and abort, so SSI requires that read-write transactions be fairly short (long-running read-only transactions may be okay). However, SSI is probably less sensitive to slow transactions than two-phase locking or serial execution.

8. The Trouble with Distributed Systems

In the end, our task as engineers is to build systems that meet user expectations, in spite of everything that goes wrong in the real world. In distributed systems, pessimism pays off, because plenty of things can go wrong:

Nondeterminism and Partial Failures

Ethernet and IP are packet-switched protocols, which suffer from queueing and thus unbounded delays in the network.
Clocks in nodes might be out of sync, suddenly jumping in time.
A node in a distributed system must assume that its execution can be paused (i.e. garbage collection) for a significant length of time at any point.

Such partial failures are inherent to distributed systems, and as such, nondeterministic. When software interacts with other nodes, there's a possibility of occasional failures, slow responses, or timeouts. Tolerance of partial failures is a goal, enabling the system to function despite broken parts.

Still, a network doesn’t guard against human error (e.g. misconfigured switches), which is a ~~major~~ cause of outages.

The bigger a system gets, the more likely it is that one of its components is broken. Over time, broken things get fixed and new things break, but in a system with thousands of nodes, it is reasonable to assume that something is always broken.

Detecting faults

Timeouts are often used, but they can't distinguish between network and node failures. Even once detected, tolerating faults is complex due to the lack of shared state between nodes. Information flows via an unreliable network, necessitating protocols that involve multiple nodes to reach agreement.

Rather than using configured constant timeouts, systems can continually measure response times and their variability (jitter), and automatically adjust timeouts according to the observed response time distribution.

With careful use of quality of service (prioritization and scheduling of packets) and admission control (rate-limiting senders), it is possible to provide statistically bounded delay.

Deliberately trigger network problems and test the system’s response (this is the idea behind Netflix's Chaos Monkey).

TCP vs UDP

Some latency-sensitive applications, such as videoconferencing, use UDP rather than TCP. It’s a trade-off between reliability and variability of delays: as UDP does not perform flow control and does not retransmit lost packets, it avoids some of the reasons for variable network delays.

Time is Money

The "MiFID II" draft European regulation for financial institutions requires all high-frequency trading funds to synchronize their clocks to within 100 microseconds of UTC, in order to help debug market anomalies such as “flash crashes” and to help detect market manipulation.

The Worst Error

Database writes can mysteriously disappear: a node with a lagging clock is unable to overwrite values previously written by a node with a fast clock until the clock skew between the nodes has elapsed. This scenario can cause arbitrary amounts of data to be silently dropped without any error being reported to the application.

In real life it's extremely difficult to calculate the true, current time. A close exception is Google’s TrueTime API, which explicitly reports the confidence interval on the local clock. When you ask it for the current time, you get back two values: [earliest, latest], which are the earliest possible and the latest possible timestamp.

Hard real-time systems

Computers that control aircraft, rockets, or cars must respond quickly to their sensor inputs. There is a specified deadline by which the software must respond; if it doesn’t, that may cause a failure of the entire system. Moreover, “real-time” is not the same as “high-performance”—in fact, real-time systems may have lower throughput, since they have to prioritize timely responses above all else.

Cost Trade-Off

You can always have a cheaper system that’s further from the ideal of a distributed system: scalable, fault tolerant, and with low latency. It depends on your situation if you choose cheap and unreliable solutions over expensive reliability.

There is no such thing as perfect reliability, but only limits of what one can realistically promise.

The internet shares network bandwidth dynamically. Senders want to act as quickly as possible, and the network switches allocates bandwidth for each packet over the wire. This approach has the downside of queueing, but the advantage is that it maximizes utilization of the wire. The wire has a fixed cost, so if you utilize it better, each byte you send over the wire is cheaper.

Multi-tenancy environments with dynamic resource partitioning provide better utilization, so it is cheaper, but it has the downside of variable delays (vs single-tenancy environments with less delay jitter).

Variable delays in networks are not a law of nature, but simply the result of a cost/benefit trade-off.

Knowledge, Truth and Lies

A service’s clients are often run by people whose priorities are very different from the priorities of the people running the service. I have been both a victim and a perpetrator of this while working for a Big Tech firm, where interests are always at conflict.

The moral of the story is that a node cannot necessarily trust its own judgment of a situation.

Byzantine Faults

Distributed systems problems become much harder if there is a risk that nodes may “lie” (send arbitrary faulty or corrupted responses)—for example, if a node may claim to have received a particular message when in fact it didn’t. Such behavior is known as a Byzantine fault.

The Byzantine Generals Problem

This problem is a generalization of the so-called Two Generals Problem, which imagines a situation in which two army generals need to agree on a battle plan. As they have set up camp on two different sites, they can only communicate by messenger, and the messenger sometimes get delayed or lost (like packets in a network).

A system is Byzantine fault-tolerant if it continues to operate correctly even if some of the nodes are malfunctioning and not obeying the protocol, or if malicious attackers are interfering with the network.

In peer-to-peer networks, where there is no such central authority, Byzantine fault tolerance is more relevant. Think of the blockchain, for example.

The theoretical description of an algorithm can declare that certain things are simply assumed not to happen—and in non-Byzantine systems, we do have to make some assumptions about faults that can and cannot happen. However, a real implementation may still have to include code to handle the case where something happens that was assumed to be impossible, even if that handling boils down to printf("Sucks to be you") and exit(666)—i.e., letting a human operator clean up the mess. (This is arguably the difference between computer science and software engineering.)

9. Consistency and Consensus

Packets can be lost, reordered, duplicated, or arbitrarily delayed in the network; clocks are approximate at best; and nodes can pause due to garbage collection or crash at any time. We need to find ways of tolerating faults in a distributed system.

The best way of building fault-tolerant systems is to find some general-purpose abstractions with useful guarantees. For example, by using a transaction, the application can pretend that there are no crashes (atomicity), that nobody else is concurrently accessing the database (isolation), and that storage devices are perfectly reliable (durability). Even though crashes, race conditions, and disk failures do occur, the transaction abstraction hides those problems so that the application doesn’t need to worry about them.

Consensus

When users register for a new social network, how can one ensure that a username is unique? If one node is going to accept a registration, it needs to somehow know that another node isn’t concurrently in the process of registering the same name. This problem leads us toward another useful transaction: consensus.

Achieving consensus means deciding something in such a way that all nodes agree on what was decided, and such that the decision is irrevocable.

Split brain 🧠

Say you have a database with single-leader replication. If the leader dies and you need to fail over to another node, the remaining database nodes can use consensus to elect a new leader. If two nodes both believe that they are the leader, we have a split brain, and it often leads to data loss.

Consistency guarantees

Most replicated databases provide at least eventual consistency (if you stop writing and wait some time, then eventually all reads will return the same value). Stronger guarantees may have worse performance or be less fault-tolerant than systems with weaker guarantees.

For example, linearizability is appealing because it makes a database behave like a variable in a single-threaded program, but it’s nevertheless slow, especially with large network delays.

Causality

Unlike linearizability, which puts all operations in a single, totally ordered timeline, causality provides us with a weaker consistency model: some things can be concurrent, so the version history is like a timeline with branching and merging. Causal consistency does not have the coordination overhead of linearizability and is much less sensitive to network problems.

Kicking the can down the road 🥫

Although a single-leader database can provide linearizability without executing a consensus algorithm on every write, it still requires consensus to maintain its leadership and for leadership changes.

Tools like ZooKeeper play an important role in providing an “outsourced” consensus, failure detection, and membership service that applications can use.

Nevertheless, not every system necessarily requires consensus: for example, leaderless and multi-leader replication systems typically do not use global consensus. The conflicts that occur in these systems are a consequence of not having consensus across different leaders, but maybe that’s okay: maybe we simply need to cope without linearizability and learn to work better with data that has branching and merging version histories.

Insights in 'Designing Data-Intensive Applications' - Part 1/3

Fri, 08 Dec 2023 00:00:00 GMT

Reading it is no easy feat (>600 pages), but I do strongly recommend it if you want to have a broader vision for the next time you have to design a system, or just want to brush up on some concepts.

Some of these notes are as they appear in the original book, most tailored by me, but all ideas are a creation of Martin or authors Martin quotes in his book.

This is Part 1. Check Part 2, Distributed Data, here, and Part 3, Derived Data, here.

Foundations of Data Systems

Who is this for? Why should I read this?

The book and this blog are for software engineers, architects, and managers who love to code and design systems. It is especially relevant if you need to make decisions about the architecture of the systems you work on. Even if you have no choice over your tools, I hope this blog will help you understand their strengths and weaknesses, just as the book did for me.

As software engineers and architects, we need to have a technically accurate and precise understanding of the various technologies and their trade-offs if we want to build good applications. Often times, aspects of a system are looked over or outright dismissed, such as message queues, caches, search indexes, frameworks for batch and stream processing.

If you understand those principles, you’re in a position to see where each tool fits in, how to make good use of it, and how to avoid its pitfalls. That’s where Kleppmann's book comes in, and that's what I want to highlight in this post.

Foundations of Data Systems

Preface

What is data-intensive anyway?

Is it yet another buzz phrase to add traction for lame LinkedIn posts? Maybe. But in essence, an application is data-intensive if data is its primary challenge—the quantity of data, the complexity of data, or the speed at which it is changing—as opposed to compute-intensive, where CPU cycles are the bottleneck.

A data-intensive application is typically built from standard building blocks that provide commonly needed functionality. For example, many applications need to:

Store data for later (databases)
Remember the result of an expensive operation, to speed up reads (caches)
Allow users to search data by keyword or filter it in various ways (search indexes
Periodically crunch a large amount of accumulated data (batch processing)
Send a message to another process, to be handled asynchronously (stream processing)

Show me the money (and the system)

Businesses need to be agile, test hypotheses cheaply, and respond quickly to new market insights by keeping development cycles short and data models flexible. No matter how strong your business plan looks, without a proper system that fits this ideal, there is little chance that startup or company will prosper.

"You’re not Google or Amazon. Stop worrying about scale!"

Building for scale that you don’t need is wasted effort and may lock you into an inflexible design. In effect, it is a form of premature optimization. However, it’s also important to choose the right tool for the job, and different technologies each have their own strengths and weaknesses.

1. Reliable, Scalable, and Maintainable Applications

An application has to meet various requirements in order to be useful. There are functional requirements (what it should do, such as allowing data to be stored, retrieved, searched, and processed in various ways), and some nonfunctional requirements (general properties like security, reliability, compliance, scalability, compatibility, and maintainability). Let's discuss the latter.

Reliability

A system is reliable if it continues to perform the correct function users expect at the desired level of performance, even in the face of adversity (hardware or software faults, users making mistakes or using the software in unexpected ways).

There are situations in which we may choose to ~~sacrifice~~ reliability in order to reduce development cost (e.g., when developing a prototype product for an unproven market) or operational cost (e.g., for a service with a very narrow profit margin)—but we should be very ~~conscious~~ of when we are cutting corners.

Scalability

A system is scalable if its performance stays ‘good enough’ under growth in data volume, traffic volume, or complexity.

In an unproven product it’s usually more important to be able to iterate quickly on product features than it is to scale to some hypothetical future load.

The architecture of systems that operate at large scale is usually highly specific to the application—there is no such thing as a one-size-fits-all scalable architecture. The problem may be the volume of reads, the volume of writes, the volume of data to store, the complexity of the data, the response time requirements, the access patterns, or (usually) some mixture of all of these plus many more issues.

Some systems can automatically add computing resources when they detect a load increase, whereas other systems are scaled manually by a human.

An architecture that is appropriate for one level of load is unlikely to cope with 10 times that load. If you are working on a fast-growing service, it is therefore likely that you will need to rethink your architecture on every order of magnitude load increase.

Maintainability

A system is maintainable if, over time, many different people will work on the system, and they are all able to work on it productively. In essence it’s about making life better for the engineering and operations teams who need to work with the system:

self-healing where appropriate, but also giving administrators manual control
good documentation and an easy-to-understand operational model

It is well known that the majority of the cost of software is not in its initial development, but in its ongoing maintenance.

Maintainability can be divided into three design principles: Operability: it should be easy to…

Keep the system running smoothly
Use monitoring for tracking down the cause of problems
Make security patches
Deploy and rollback
Migrate applications
Preserve the organization's knowledge (often looked over)
Provide visibility into the runtime behavior
Avoiding dependency on individual machines
Plan the expected capacity of your systems, and monitor for changes

Good operations can often work around the limitations of bad (or incomplete) software, but good software cannot run reliably with bad operations.

Simplicity

Make it easy for new engineers to understand the system, by removing as much complexity as possible from the system. (This is not the same as simplicity of the user interface).
Complexity slows down everyone who needs to work on the system, further increasing the cost of maintenance.

Evolvability

Make it easy for engineers to make changes to the system in the future.

A tool for the task

Many new tools for data storage and processing have emerged in recent years. They are optimized for a variety of different use cases, and they no longer neatly fit into traditional categories. For example, there are data stores that are also used as message queues (Redis), and there are message queues with database-like durability guarantees (Apache Kafka).

This ambiguity can work for specific use cases, but generally makes the choice of a tool harder and sticks your system to the tools limits, opposite to the more flexible approach, where the work is broken down into tasks that can be performed efficiently on a single tool, and those different tools are stitched together using application code.

Faults and failures

A fault is usually defined as one component of the system deviating from its spec, whereas a failure is when the system as a whole stops providing the required service to the user.

Faults can be in hardware (typically random and uncorrelated), software (bugs are typically systematic and hard to deal with), and humans (who inevitably make mistakes from time to time). Fault-tolerance techniques can hide certain types of faults from the end user

Hard disks are reported as having a mean time to failure (MTTF) of about 10 to 50 years. On a storage cluster with 10,000 disks, we should expect on average one disk to die per day.

It is impossible to reduce the probability of a fault to zero; it is usually best to design fault-tolerance mechanisms that prevent faults from causing failures. Failures of e-commerce sites can have huge costs in terms of lost revenue and damage to reputation (I was involved in a crash that cost my employer $40K in only 3 hours).

It is a common practice in big tech companies, such as Netflix, Amazon or MercadoLibre, to deliberately induce faults, so as to ensure that the fault-tolerance machinery is continually exercised and tested. A famous one is Netflix's Chaos Monkey.

Such fault-tolerant systems also have operational advantages: a single-server system requires planned downtime if you need to reboot the machine (to apply operating system security patches, for example), whereas a system that can tolerate machine failure can be patched one node at a time, without downtime of the entire system.

If a system is expected to provide some guarantee (for example, in a message queue, that the number of incoming messages equals the number of outgoing messages), it can constantly check itself while it is running and raise an alert if a discrepancy is found.

Expect the best, prepare for the worst

Design systems in a way that minimizes opportunities for error. For example, well-designed abstractions, APIs, and admin interfaces make it easy to do “the right thing” and discourage “the wrong thing.” However, if the interfaces are too restrictive people will work around them, negating their benefit, so this is a tricky balance to get right.
Provide fully featured non-production sandbox environments where people can explore and experiment safely, using real data, without affecting real users.
Set up detailed and clear monitoring, such as performance metrics and error rates. When a problem occurs, metrics can be invaluable in diagnosing the issue: requests per second to a web server, the ratio of reads to writes in a database, the number of simultaneously active users in a chat room, the hit rate on a cache, or something else.

Twitter example

Post tweet: a user can publish a new message to their followers (4.6k requests/sec on average, over 12k requests/sec at peak).
Home timeline: a user can view tweets posted by the people they follow (300k requests/sec).

Twitter’s scaling challenge is not primarily due to tweet volume, but due to fan-out: each user follows many people, and each user is followed by many people. When a user posts a tweet, the system has to look up all the people who follow that user, and insert the new tweet into each of their home timeline caches. The request to read the home timeline is then cheap, because its result has been computed ahead of time.

This works better because the average rate of published tweets is almost two orders of magnitude lower than the rate of home timeline reads, and so in this case it’s preferable to do more work at write time and less at read time. But some users have over 30 million followers. This means that a single tweet may result in over 30 million writes to home timelines! Doing this in a timely manner—Twitter tries to deliver tweets to followers within five seconds—is a significant challenge.

Post tweet

Tweets from any celebrities that a user may follow are fetched separately and merged with that user’s home timeline when it is read. This hybrid approach is able to deliver consistently good performance.

Measuring Performance: response time percentiles

When you increase a load parameter and keep the system resources (CPU, memory, network bandwidth, etc.) unchanged, how is the performance of your system affected?
When you increase a load parameter, how much do you need to increase the resources if you want to keep performance unchanged?

Both questions require performance numbers.

In a batch processing system such as Hadoop, we usually care about throughput—the number of records we can process per second, or the total time it takes to run a job on a dataset of a certain size. In online systems, what’s usually more important is the service’s response time.

Latency and response time are often used synonymously, but they are not the same. The response time is what the client sees: besides the actual time to process the request (the service time), it includes network delays and queueing delays. Latency is the duration that a request is waiting to be handled—during which it is latent, awaiting service.

We therefore need to think of response time not as a single number, but as a distribution of values that you can measure. The mean is not a very good metric if you want to know your “typical” response time, because it doesn’t tell you how many users actually experienced that delay.

If your median response time is 200ms, that means half your requests return in less than 200ms, and half your requests take longer than that. The median is also known as the 50th percentile, and sometimes abbreviated as p50. If the 95th percentile response time is 1.5 seconds, that means 95 out of 100 requests take less than 1.5 seconds.

Amazon describes response time requirements for internal services in terms of the 99.9th percentile, even though it only affects 1 in 1,000 requests. This is because the customers with the slowest requests are often those who have the most data on their accounts because they have made many purchases—that is, they’re the most valuable customers!

You may want to keep a rolling window of response times of requests in the last 10 minutes.

Bad effects

As a server can only process a small number of things in parallel (limited, for example, by its number of CPU cores), it only takes a small number of slow requests to hold up the processing of subsequent requests—an effect sometimes known as head-of-line blocking. Therefore, it is important to measure response times on the client side.

Even if only a small percentage of backend calls are slow, the chance of getting a slow call increases if an end-user request requires multiple backend calls, and so a higher proportion of end-user requests end up being slow (an effect known as tail latency amplification).

2. Data Models and Query Languages

The best-known data model today, proposed by Edward Codd in 1970, is data organized into relations (tables in SQL), where each relation is an unordered collection of tuples (rows in SQL). The typical use cases for mainframes in the 1960s and 1970s were transaction processing (sales transactions, stock-keeping in warehouses) and batch processing (customer invoicing, payroll).

Decades later, new use cases and changes in scale brought new solutions to the space.

The Birth of NoSQL

NoSQL is the latest attempt to overthrow the relational model’s dominance. The name “NoSQL” is unfortunate, since it doesn’t actually refer to any particular technology—it was originally intended simply as a catchy Twitter hashtag for a meetup on open source, distributed, non-relational databases in 2009.

There are several driving forces behind the adoption of NoSQL databases, including:

A need for greater scalability: very large datasets or write throughput
A preference for free and open source over commercial products
Specialized query operations that are not well supported by the relational model
Frustration with the restrictiveness of relational schemas

The Object-Relational Impedance Mismatch

Picture the database model of a profile on LinkedIn. It may have multiple one-to-many relations: one user can have many positions, many education entries and many contact info options. This translates into a separate table for each relation, and into multiple joins when constructing the entire profile.

A JSON representation of this profile has better locality than this multi-table schema. If you want to fetch a profile in the relational example, you need to either perform multiple queries or perform a messy multi-way join between the users table and its subordinate tables. In the JSON representation, all the relevant information is in one place, and one query is sufficient.

ID or text? A question of duplication

When you use an ID, the information that is meaningful to humans (millions of users can share the same education entry) is stored in only one place, and everything that refers to it uses an ID (which only has meaning within the database). When you store the text directly, you are duplicating the human-meaningful information in every record that uses it. Anything that is meaningful to humans may need to change sometime in the future—and if that information is duplicated, all the redundant copies need to be updated. That incurs write overheads, and risks inconsistencies. Removing such duplication is the key idea behind normalization in databases.

Unfortunately, normalizing this data requires many-to-one relationships (many people live in one particular region, many people work in one particular industry), which don’t fit nicely into the document model. In relational databases, it’s normal to refer to rows in other tables by ID, because joins are easy. In document databases, joins are not needed for one-to-many tree structures, and support for joins is often weak.

Moreover, even if the initial version of an application fits well in a join-free document model, data has a tendency of becoming more interconnected as features are added to applications.

What to choose? It depends...

The main arguments in favor of the document data model are schema flexibility, better performance due to locality, and that for some applications it is closer to the data structures used by the application. The relational model counters by providing better support for joins, and many-to-one and many-to-many relationships.

If your application does use many-to-many relationships, the document model becomes less appealing. It’s possible to reduce the need for joins by denormalizing, but then the application code needs to do additional work to keep the denormalized data consistent.

Schemaless?

No schema means that arbitrary keys and values can be added to a document, and when reading, clients have no guarantees as to what fields the documents may contain.

Document databases are sometimes called schemaless, but that’s misleading, as the code that reads the data usually assumes some kind of structure. A more accurate term is schema-on-read (the structure of the data is implicit, and only interpreted when the data is read), in contrast with schema-on-write (the traditional approach of relational databases, where the schema is explicit and the database ensures all written data conforms to it).

Schema-on-read is similar to dynamic (runtime) type checking in programming languages, whereas schema-on-write is similar to static (compile-time) type checking.

On the one hand, changes to the the structure of data in a document database have to be added to application code to handle the old and new documents. On the other hand, in a “statically typed” database schema, you would typically perform a migration, which often requires downtime and is slow. In cases where all records are expected to have the same structure, schemas are a useful mechanism for documenting and enforcing that structure. Otherwise, the schema-on-read approach is more advantageous.

Storage Locality

A document is usually stored as a single continuous string, encoded as JSON for example. If your application often needs to access the entire document, there is a performance advantage to this storage locality. This advantage only applies if you need large parts of the document at the same time. The database typically needs to load the entire document, even if you access only a small portion of it, which can be wasteful on large documents.

Graph Stuff

What if many-to-many relationships are very common in your data? The relational model can handle simple cases of many-to-many relationships, but as the connections within your data become more complex, it becomes more natural to start modeling your data as a graph.

Social graphs Vertices are people, and edges indicate which people know each other.

The web graph Vertices are web pages, and edges indicate HTML links to other pages.

Graphs are also good for evolvability: as you add features to your application, a graph can easily be extended to accommodate changes in your application’s data structures.

Graph Queries

In a relational database, you usually know in advance which joins you need in your query. In a graph query, you may need to traverse a variable number of edges before you find the vertex you’re looking for—that is, the number of joins is not fixed in advance.

If the same query can be written in 4 lines in one query language but requires 29 lines in another, that just shows that different data models are designed to satisfy different use cases. It’s important to pick a data model that is suitable for your application.

3. Storage and Retrieval

In order to tune a storage engine to perform well on your kind of workload, you need to have a rough idea of what the storage engine is doing under the hood. There is a big difference between storage engines that are optimized for transactional workloads and those optimized for analytics.

An Important Trade-Off: writes

Well-chosen indexes speed up read queries, but every index slows down writes. You can then choose the indexes that give your application the greatest benefit, without introducing more overhead than necessary (i.e. indexes must fit in memory).

Some indexes, like log-structured indexes, rewrite data multiple times due to their repeated compaction and merging nature. This effect—one write to the database resulting in multiple writes to the disk—is known as write amplification, and has a direct performance cost.

Another type of indexes, B-trees, are very ingrained in the architecture of databases and provide consistently good performance for many workloads. In new datastores, log-structured indexes are becoming increasingly popular. There is no quick and easy rule for determining which type of storage engine is better for your use case, so it is worth testing empirically.

Another Trade-Off: clustered indexes

A compromise between a clustered index (storing all row data within the index) and a non-clustered index (storing only references to the data within the index) is known as a covering index or index with included columns.

As with any kind of duplication of data, clustered and covering indexes can speed up reads, but they require additional storage and can add overhead on writes. Databases also need to go to additional effort to enforce transactional guarantees, because applications should not see inconsistencies due to the duplication.

Multi-column indexes

In a database of weather observations you could have a two-dimensional index (on date, temperature) in order to efficiently search for all the observations during the year 2013 where the temperature was between 25 and 30C. With a one-dimensional index, you would have to either scan over all the records from 2013 (regardless of temperature) and then filter them by temperature, or vice versa. A 2D index could narrow down by timestamp and temperature simultaneously!

A question of cost and encoding

As RAM becomes cheaper, the cost-per-gigabyte argument in favor of disks is eroded. Many datasets are simply not that big, so it’s quite feasible to keep them entirely in memory, potentially distributed across several machines. This has led to the development of in-memory databases.

Some in-memory key-value stores, such as Memcached, are intended for caching use only, where it’s acceptable for data to be lost if a machine is restarted. But other in-memory databases aim for durability, which can be achieved with special hardware (such as battery-powered RAM), by writing a log of changes to disk or snapshots to disk.

Counterintuitively, the performance advantage of in-memory databases is not due to the fact that they don’t need to read from disk. Even a disk-based storage engine may never need to read from disk if you have enough memory, because the operating system caches recently used disk blocks in memory anyway. Rather, they can be faster because they can avoid the overheads of encoding in-memory data structures in a form that can be written to disk.

Transaction processing vs Analytic systems

A transaction needn’t necessarily have ACID properties. Transaction processing just means allowing clients to make low-latency reads and writes—as opposed to batch processing jobs, which only run periodically.

Property	Transaction processing systems (OLTP)	Analytic systems (OLAP)
Main read pattern	Small number of records per query, fetched by key	Aggregate over large number of records
Main write pattern	Random-access, low-latency writes from user input	Bulk import (ETL) or event stream
Primarily used by	End user/customer, via web app	Internal analyst, for decision support
What data represents	Latest state of data (current point in time)	History of events that happened over time
Dataset size	Gigabytes to terabytes	Terabytes to petabytes

In the early 1990s, there was a trend for companies to stop using their OLTP systems for analytics purposes, and to run the analytics on a separate database instead. This separate database was called a data warehouse.

Data Warehousing

OLTP systems (i.e. checkout, inventory, etc) are expected to be highly available and to process transactions with low latency, since they are critical for business. It's a bad idea to let business analysts run ad hoc analytic queries on an OLTP database, since those queries are often expensive, which can harm the performance of concurrently executing transactions.

A data warehouse, by contrast, is a separate database that analysts can query without affecting OLTP operations. It contains read-only copy of the data in all the various OLTP systems in the company.

Data Warehousing

Even though data warehouse and OLTP databases share an SQL interface, they differ in their query optimizations, leading to specialization for transaction processing or analytics workloads by most vendors (but not both).

Separate table of dates

Date and time are often represented using dimension tables (a subset of fact tables), because this allows additional information about dates (such as public holidays) to be encoded, allowing queries to differentiate between sales on holidays and non-holidays.

Columnar Storage

When your queries require sequentially scanning across a large number of rows, it becomes important to encode data very compactly, to minimize the amount of data that the query needs to read from disk.

The idea behind column-oriented storage is simple: don’t store all the values from one row together, but store all the values from each column together instead. If each column is stored in a separate file, a query only needs to read and parse those columns that are used in that query, which can save a lot of work.

Column storage is easiest to understand in a relational data model, but it applies equally to non-relational data. For example, Parquet is a columnar storage format that supports a document data model, based on Google’s Dremel.

Data^Data^Data: Data Cubes

A common special case of a materialized view is known as a data cube or OLAP cube.

A materialized view is an actual copy of query results written to disk, unlike a virtual view, which is a shortcut for writing queries, as the SQL engine expands it into the underlying query on-the-fly when read.

The advantage of a materialized data cube is that certain queries become very fast because they have effectively been precomputed. For example, if you want to know the total sales per store yesterday, you just need to look at the totals along the appropriate dimension—no need to scan millions of rows.

The disadvantage is that a data cube doesn’t have the same flexibility as querying the raw data. For example, there is no way of calculating which proportion of sales comes from items that cost more than $100, because the price isn’t one of the dimensions. Most data warehouses therefore try to keep as much raw data as possible, and use aggregates such as data cubes only as a performance boost for certain queries.

4. Encoding and Evolvability

Scenarios in which data encodings are important

Databases, where the process writing encodes the data and the process reading decodes it
RPC and REST APIs, where the client encodes a request, the server decodes it and encodes a response, and the client finally decodes the response
Asynchronous message passing (using message brokers), where nodes communicate by sending each other messages that are encoded by the sender and decoded by the recipient

A bit of care

Relational databases generally assume that all data in the database conforms to one schema at any one point in time. By contrast, schema-on-read (“schemaless”) databases don’t enforce a schema, so the database can contain a mixture of older and newer data formats written at different times.

With server-side applications you may want to perform a rolling upgrade of nodes. This allows new versions to be deployed without service downtime, and thus encourages more frequent releases and better evolvability.
With client-side applications you’re at the mercy of the user, who may not install the update for some time!

Backward compatibility Newer code can read data that was written by older code.

Forward compatibility Older code can read data that was written by newer code.

The JSON returned by Twitter’s API includes tweet IDs twice, once as a JSON number and once as a decimal string, to work around the fact that the numbers are not correctly parsed by JavaScript applications

Formats for Encoding Data

Programs usually work with data in two different representations:

In memory, data is kept in objects, structs, lists, arrays, hash tables, trees, and so on. These data structures are optimized for efficient access and manipulation by the CPU (typically using pointers).
When you want to write data to a file or send it over the network, you have to encode it as some kind of self-contained sequence of bytes (for example, a JSON document). Since a pointer wouldn’t make sense to any other process, this sequence-of-bytes representation looks quite different from the data structures that are normally used in memory.

Don't use your language's built-in encoding

If an attacker can get your application to decode an arbitrary byte sequence, they can instantiate arbitrary classes.
Efficiency (CPU time taken to encode or decode, and the size of the encoded structure) is also often an afterthought. For example, Java’s built-in serialization is notorious for its bad performance and bloated encoding.

Revelatory Facts in 'Why We Sleep'

Sun, 03 Sep 2023 00:00:00 GMT

Matthew Walker’s book, “Why We Sleep,” is a real game-changer. All the cool stuff I’m sharing comes straight from him or the authors he quotes. And hey, I might’ve rephrased some things to make it easier to understand.

Turns out, sleep is super crucial for everything we do! If we don’t get enough shut-eye, our thinking gets fuzzy, and our memory goes bust. Plus, it messes with our health big time, making us more prone to all sorts of problems.

Let me me share with you the facts and practices that most caught my attention after reading the book, and, in the last part, I will talk about some insights about my last 4 years of sleep (hint: they are not good).

Most of the sentences in this text were extracted and readapted. All ideas and quotes are found on the pages of the book.

Part 1: This Thing Called Sleep
Part 2: Why Should You Sleep?
Part 3: How and Why We Dream
- C9: Routinely Psychotic
Part 4: From Sleeping Pills to Society Transformed
- C15: Sleep and Society
  - Sleep and education
  - Sleep and Health Care
- C16: A New Vision for Sleep in the Twenty-First Century
  - Public Policy and Societal Change
My Data: What can I learn from 4 years of sleep tracking?
Farewell

Part 1: This Thing Called Sleep

C1: To Sleep

Routinely sleeping less than six hours demolishes your immune system. This affected lifestyle increases the likelihood you will develop Alzheimer’s disease, blood sugar level disruptions, pre-diabetes, blocked arteries, and multiple other complications.

The WHO even declared a sleep loss epidemic throughout industrialized nations— nevermind coronavirus.

Sleep is no joke. After all, one third of our entire lives will be spent sleeping. That’s about 25 years out of a life expectancy of 75. 25 years! And each second is very much needed! But it is puzzling in its own way. While we sleep, we cannot reproduce, nurture, protect ourselves or others, work, study, laugh, cry, nothing albeit sleep. It’s a time of complete vulnerability.

Still, the benefits are unavoidable to our brains and bodies. Sleeping allows us to navigate next-day social and psychological challenges with cool-header composure. I imagine you can notice the difference in your mood and sharpness whenever you went through hours of no sleep.

This are not shiny, unfounded words. Science is increasingly experimenting and observing the consequences of irregular, bad sleep. By studying sleep wave recordings, researchers can potentially diagnose which type of dementia a sleep deprived person can develop.

C2: Caffeine, jet lag, and melatonin: losing and gaining control of your sleep rhythm

I have always been an owl. That is, I am the owlness chronotype, not the larkness one. Whether you are a night owl like me or a morning lark, it is believed to be strongly linked to genetics (you probably share any with one or both your parents).

Sadly, society (and parents most of the time) treats labels us as lazy, since we won’t wake up early in the day. Morning larks go to the extent of criticizing owls, citing their lack of ‘willingness and discipline’, as if owls deliberately chose such chronotype. This is unfair and wrong: night owls are not such by choice, but by genetics. They are bound to a delayed schedule by unavoidable DNA hardwiring. It is not the owls’ conscious fault, but their genetic fate.

Besides, think about this trait in terms of survival. What if we were all owls or all larks? We would all leave a window of time were we are fully asleep, and thus completely vulnerable to threats and attacks. Mother Nature seems to have triggered a smarted move. By spreading us between the two chronotypes, we increase our survival fitness, with ‘guards’ on call on every hour.

Another interest comment is that on the difference we experience when travel either west or east. When we travel westward, in the direction of our internal clock (our circadian rhythm), that “day” is longer than 24 hours and it feels easier to accommodate to. However, if we travel eastward, the “day” seems shorter, goes against our clock, and thus feels rather harder to adapt to.

Watchout for caffeine consumption. By hijacking and occupying these receptors, caffeine blocks the sleepiness signal normally communicated to the brain by adenosine. The upshot: caffeine tricks you into feeling alert and awake, despite the high levels of adenosine that would otherwise seduce you into sleep.

Caffeine has an average half-life of 5 to 7 hours. Let's say that you have a cup of coffee after your evening dinner, around 7:30 PM. This means that by 1:30 AM, 50% of the caffeine may still be active and circulating throughout your brain tissue. In other words, by 1:30 AM, you're only halfway to completing the job of cleansing your brain of the caffeine you drink after dinner.

Also be aware that de-caffeinated does not mean non-caffeinated. One cup of decaf usually contains 15% to 30% of the dose of a regular cup of coffee, which is far from caffeine free. Should you drink 3 to 4 cups of decaf in the evening, it is just as damaging to your sleep as one regular cup of coffee.

Two Factors

Setting aside the extreme case of sleep deprivation, how do you know that are your routine getting enough sleep? While a sleep assessment is needed to thoroughly address this issue, an easy rule of thumb is to answer two simple questions. First, after waking up in the morning, could you fall back asleep at 10 or 11 AM? If the answer is “yes”, you are likely not getting sufficient sleep quantity and/or quality. Second, can you function optimally without caffeine before noon? If the answer is “no”, then you are most likely self-medicating your state of chronic sleep deprivation.

C3: Defining and generating sleep

The two stages of sleep NREM and REM play out in a recurring push-pull battle of brain domination across the night. The cerebral war between the two is one and lost every 90 minutes, rolled first by NREM sleep, followed by the comeback of REM sleep.

Architecture of sleep

In the first half of the night, the vast majority of our 90 minute cycles are consumed by deep NREM sleep, and very little by REM sleep, as can be seen in cycle one of the figure above. But as we transition through into the second half of the night, this seesaw balance shifts, with most of the time dominated by REM sleep, with little, if any, deep NREM sleep. Cycle 5 is a perfect example of this REM-rich type of sleep.

A key function of deep NREM sleep, which predominates early in the night, is to do the work of weeding out and removing unnecessary neural connections. In contrast, the dreaming stage of REM sleep, which prevails later in the night, plays a role in strengthening those connections.

When you decide to sleep full eight hours of sleep starting from midnight but you decide to wake up at 6 AM instead of 8 AM you will lose 60 to 90% of all your REM sleep, even though you are losing 25% of your total sleep time.

NREM sleep is one of the most epic display of neural coordination that we know of. Many thousands of brain cells have all decided to unite and fire at the same time.

Each night, the long range brain waves of deep sleep will move memory packets (recent experiences) from short-term storage site, which is fragile, to a more permanent, and thus safer, long-term storage location.

When it comes to information processing, think of the wake state principally as reception, NREM sleep as reflection, and REM sleep as integration.

By eliminating muscle activity during REM sleep you are prevented from acting out your dream experience. Wise, then, of Mother Nature to have tailored a physiological straight jacket that forbids fictional movements from becoming a reality, specially considering that you have stopped consciously perceiving your surroundings.

C4: Ape beds, dinosaurs, and napping with half a brain: who sleeps, how do we sleep, and how much?

Theodosius Dobzhansky: Nothing in biology makes sense except in light of evolution

Considering how biologically beneficial the state of sleep is, we can pose a very different theory: sleep was the first stage of life on this planet, and it was from sleep that wakefulness emerged. It may be a preposterous hypothesis, and one that nobody is taking seriously or exploring, but I personally do not think it to be entirely unreasonable.

As millennia unfolded and evolution crowned its current accomplishment with the genesis of the brain, the demand for sleep only increased, tending to the needs of this most precious of all physiological apparatus. It can be suggested that one evolutionary function that demands more also is the need to service an increasingly complex nervous system.

To dream or not to dream

Insects, fish and most reptiles show no queer signs of REM sleep. Only birds and mammals have full-blown REM sleep. It suggests that dream (REM sleep) is the new kid on the evolutionary block. REM sleep seems to have emerged to support functions that are NREM sleep alone could not accomplish, or that REM sleep was more efficient at accomplishing.

It is important to note that regardless of the amount of recovery opportunity, the brain never comes close to getting back all the sleep it has lost. This is true for total sleep time, just as it is for NREM sleep and for REM sleep. That humans can never “sleep back” that which we have previously lost is one of the most important take-homes of this book, the saddening consequences of which I will describe in chapter 7 and 8.

Cetaceans, such as dolphins and whales, have sleep which is only NREM, which can be unihemispheric, meaning they will sleep with half of the brain at a time! One half always stays awake to maintain life necessary movement, but the other half of the brain will fall into NREM sleep.

Even more, in-flight migrating birds will grab remarkably brief periods of sleep lasting only seconds in duration. This ultra power naps are just sufficient to avert the ruinous brain and body deficits that what otherwise ensure from prolonged total sleep deprivation.

How should we sleep

Hunter-gatherer tribes, whose way of life has changed little over the past thousands of years, sleep in a biphasic pattern. These groups take a similarly longer to sleep period at night, greater than the average of Western cultures, filled by a 30 to 60 minute nap in the afternoon.

Not enjoying a full afternoon nap, further contributes to our state of sleep bankruptcy.

The practice of biphasic sleep is not cultural in origin, it is deeply biological. All humans have a genetically hardwire dip in alertness that occurs in the midafternoon hours. We all experience a brief dissent from high degree wakefulness to low level alertness after the midday meal. Should you ever have to give a presentation at work, for your own sake – and then of the conscious state of your listeners – if you can, avoid the afternoon slot!

There is genetic evidence that the true painter of biphasic sleep consists of continuous sleep at night, followed by a shorter midafternoon nap.

A study made in more than 23,000 Greek adults consisted of tracking the group across a six-year period as the siesta practice came to an end for many of them.

Does that abandoned regular siestas went on to suffer at 37% increase risk of death from heart disease across the six year period, relative to those who maintained regular daytime naps. The effect was specially strong in workingmen, were the ensuing mortality risk of not napping increased by well over 60%.

It is perhaps surprising that in the small enclaves of Greece were siestas still remain intact, men are nearly 4 times as likely to reach the age of 90 as American males.

NREM sleep helps transfer and make safe newly learned information into long-term storage sites of the brain. But it is REM sleep that takes these freshly minted memories and begins colliding them with the entire back catalogue of your life’s autobiography.

REM sleep dreaming therefore represents tenable new contributing factor, among others, that led to our astonishingly rapid evolutionary rise to power, for better and worse, a new sleep-fueled, globally dominant social superclass!

C5: Changes in sleep across the life span

Sleep after birth

Depriving infants rats of REM sleep stalled construction of their neural rooftop-the cerebral cortex of the brain. Day after day, the half-finished roofline of the sleep-starved cerebral cortex shows no growth change.

Infants and young children who show signs of autism, or who are diagnosed with autism, do not have normal sleep patterns or amounts. The circadian rhythms of autistic children are also weaker than their non-autistic counterparts, showing a flatter profile of melatonin across the 24 hour period rather than a powerful rise in concentration at night and rapid fall throughout the day.

Autistic individuals show a 30% to 50% deficit in the amount of REM sleep they obtain, relative to children without autism. Irrespective of casuality issues, tracking sleep abnormalities represents a new diagnostic hope for the early detection of autism.

The newborns of heavy drinking mothers spent far less time in the active state of REM sleep compared with infants of similar age but who were born of mothers who did not drink during pregnancy. Besides, infants of heavy drinking mothers showed a 200% reduction of vibrant electrical activity relative to the infants born of non-alcohol-consuming mothers. When babies consume alcohol laced milk, their sleep is more fragmented, they spend more time awake, and they suffer a 20 to 30% suppression of REM sleep soon after.

Sleep and adolescence

Even though the degree of neural networks connectivity decreases during development, the physical size of our brain cells, and thus the physical size of the brain and head, increases. Adolescents have a less rational version of an adult brain, one that takes more risks and has relatively poor decision making skills.

Deep sleep may be a driving force of brain maturation, not the other way around. Rationality is one of the last things to flourish in teenagers, as it is the last brain territory to receive sleep’s maturational treatment.

The circadian rhythm of a teenager will not be miraculously coaced into a change. Asking that same teenager to wake up at seven the next morning and function with intellect, grace, and good mood is the equivalent of asking you, their parent, to do the same at 4 or 5 AM.

Passing into your mid- and late 40s, age will have stripped you of 60% to 70% of the deep sleep you were enjoying as a young teenager. By the time you reach 70 years old, you will have lost 80 to 90% of your youthful deep sleep.

Many seniors progress through their later years not fully realizing how degraded their deep sleep quantity and quality have become. This is an important point: it means that elderly individuals fail to connect their deterioration in health with their deterioration in sleep, despite casual links between the two having been known to scientists for many decades.

Any individual, no matter what age, will exhibit physical ailments, mental health instability, reduced alertness, and impaired memory if their sleep is chronically disrupted.

The third sleep change with advanced age is that of circadian timing. In sharp contrast to adolescents, seniors commonly experience a regression in sleep timing, leading to earlier and earlier bedtimes. The cause is an earlier evening release and peak of melatonin as we get older, instructing an earlier start time for sleep. Restaurants in retirement communities have long known of this age related shift in bed time preference, epitomized and accommodated by the “early bird special”. Their regressed circadian rhythm, instructed by an earlier release of melatonin, left them no choice.

Plentiful later-afternoon daylight will help the delay the evening release of melatonin, helping push the timing of sleep to a later hour. Make sure to wear sun protection of some sort, such as a hat, but leave the sunglasses at home.

Part 2: Why Should You Sleep?

C6: Your Mother and Shakespeare Knew

Observations of the brain show a strikingly reliable loop of electrical current pulsing throughout the brain that repeats every 100 to 200 ms.

This electrical transaction in the quiet secrecy of sleep is one that shifts fact-based memories from the temporary storage depot, the hippocampus, to a long-term secure vault, the cortex. This shift is the product of slow brainwaves of deep NREM sleep that serve as a courier service. In doing so, sleep helps future-proof those memories.

Studies shows a memory retention benefit of between 20% and 40% being offered by sleep, compared to the same amount of time awake.

Not only does NREM sleep maintain those memories you have successfully learned before bed, but it will even salvage those that appear to have been lost soon after learning.

Another study shows that relative to a control group that slept but had no synchronous auditory chimes at night, the auditory stimulation increased the power of the slow brainwaves and returned an impressive 40% memory enhancement the next morning

Sleep to forget?

The capacity to forget can, in certain contexts, be as important as the need for remembering. In this way, sleep helps you retain everything you need and nothing that you don’t.

This occurs when taking a nap while studying for an exam, for instance. The more of those NREM sleep spindles a person has during a nap, the greater the efficiency with which they strengthen concepts tagged for remembering and actively eliminate those designated for forgetting.

Practice alone does not make perfect

Muscle memory is, in fact, brain memory. The routine itself, the memory program, resides firmly and exclusively within the brain.

A pianist friend of the author told him that he would be practicing a particular piece for hours and wouldn’t be able to master it. Often, he would make the same mistake at the same place in a particular movement. He would then go to bed frustrated, but the next morning, after sitting back down at the piano, he would just play it perfectly.

Does this mean that the most repeated and entrusted teaching edict, “practice makes perfect“ is violated? Perhaps it was practice, with sleep, that makes perfect?

Perhaps more relevant to the modern world is the time-of-night that was discovered. Those last two hours of sleep are precisely the window that many of us feel it is OK to cut short to get a jump start on the day. As a result, we miss out on this feast of late morning sleep spindles that are deeply linked with the offline memory boost. Loose out on the last hours of sleep for an 8 hour night, and your memory capacity can be seriously affected.

Sleep and Sport

Some teams are aware of sleep’s importance before a game. But sleep after a game can be as essential. Post-performance sleep accelerates physical recovery from common inflammation, stimulates muscle repair, and helps restock cellular energy in the form of glucose and glycogen.

Sleep for Creativity

Sleep provides a nighttime theater in which your brain tests out and builds connections between vast stores of information. This task is accomplished using a bizarre algorithm that is biased towards seeking out the most distant, non-obvious associations, rather like a backward Google search.

C7: Too extreme for the Guinness book of world records

Sleep deprivation and the brain

Most worrying from a societal perspective were the individuals in the group who obtained six hours of sleep a night—something that may sound familiar to many of you. Ten days of six hours of sleep a night was all it took to become as impaired in performance as going without sleep for 24 hours straight.

You do not know how sleep-deprived you are when you are sleep-deprived

With chronic sleep restriction over months or years, an individual will actually acclimate to their impaired performance, lower alertness, and reduced energy levels.

60 years of scientific research prevents the author from accepting anyone who tells him that he or she can “get by on just four or five hours of sleep a night just fine“.

And I can also say this. I had teachers at school who showed off how little they slept. This different influenced me in believing that sleep was an obstacle to one’s progress. Even into my university years, I always compromised sleep over extra study time, with questionable results.

Even though one might believe that sleep hours can be recovered during the weekend, studies show that the brain is in capable of that. It is simply not enough to repair the damage done by a work week of sleep deprivation.

Operating on less than five hours of sleep and your risk of a car crash increases threefold. Get behind the wheel of a car when having slept just four hours or less the night before and you are 11.5 times more likely to be involved in a car accident.

The recycle rate of a human being is around 16 hours. After 16 hours of being awake, the brains begins to fail. Humans need more than seven hours of sleep each night to maintain competitive performance. After 10 days of just seven hours of sleep, the brain is as dysfunctional as it would be after going without sleep for 24 hours. Three full nights of recovery sleep are insufficient to restore performance back to normal levels after a week of short sleeping. Besides, the human mind cannot actually sense how sleep deprived it is when sleep deprived.

Drowsy driving alone is worse than driving drunk. Drunk drivers are often late in breaking, and late in making evasive maneuvers. But when you fall asleep, or have a microsleep, you stop reacting altogether!

With a full night of plentiful sleep we have a balanced mix between our emotional gas pedal (amygdala) and break (prefrontal cortex). Without sleep, however, the strong coupling between these two brain regions is lost. We cannot rein in our impulses -too much emotional gas pedal and not enough regulatory break. Without the rational control giving to us each night by sleep, we are not on a neurological and emotional balance.

There is no psychiatric condition in which sleep is normal. This is true of depression, anxiety, PTSD, schizophrenia, and bipolar disorder. Studies show that improving sleep quantity, quality, and regularity heals the mind of numerous psychiatric populations.

A lack of sleep is a deeply penetrating and corrosive force that enfeebles the memory making apparatus within your brain, preventing you from constructing lasting memory traces. It is rather like building a sand castle too close to the tide line—the consequences are inevitable.

It is clear now that pulling all nighter‘s is not a wise idea for learning, not because you won’t learn during those hours, but because you won’t recall later on what you learned previously!

This, of course, is really not under the consideration of faculties around the globe. Teachers accumulate exams in the final days of a semester, leaving no other choice to students than to compromise sleep hours in favor of learning.

Many can answer to that argument saying that the students “make their own choices” or that students “are responsible if they do not sleep properly”. But science shows something different. If you don’t sleep the very first night after learning, you lose the chance to consolidate those memories, even if you get lots of “catch up“ sleep thereafter. Sleep is not like the bank. You cannot accumulate a debt and hope to pay it off at a later point in time. Sleep for memory consolidation is an all or nothing event. It is a concerning result in our 24 seven, hurry up, don’t wait society.

Sleep and Alzheimer’s disease

As we get older, it is common for us to have a general loss of deep sleep. But for Alzheimer’s patients, there is a ruthless eroding of the very deepest of the powerful slow brain waves of NREM sleep. This distinction is important, since it means that the sleep impairment caused by amyloid buildup in the brain is more than just “normal aging“.

Patients with a high amyloid deposits in the frontal regions of the brain have the most severe loss of deep sleep. As a consequence, they fail to successfully consolidate new memories. Overnight forgetting, rather than remembering, takes place. The disruption of deep NREM sleep is therefore a hidden middleman brokering the bad deal between amyloid and memory impairment in Alzheimer’s disease.

Should you experimentally prevent a mouse from getting NREM sleep, keeping it awake instead, there is an immediate increase in amyloid deposits within the brain. Without sufficient sleep, amyloid plaques buildup in the brain, specially in deep sleep generating regions, attacking and degrading them. The loss of deep and NREM sleep caused by this assault therefore lessens the ability to remove amyloid from the brain at night, resulting in greater amyloid deposition. More amyloid, less deep sleep, less deep sleep, more amyloid, and so on and so forth.

C8: Cancer, heart attacks, and a shorter life

Sleep deprivation and the body

Over a 14 year period, those sleeping six hours or less were 400% or 500% more likely to suffer one or more cardiac arrests than those sleeping more than six hours. This relationship between short sleep and a heart failure remains strong even after controlling for other known cardiac risk factors, such as smoking or physical activity.

Like a car engine that is revved to shrieking extreme for sustained periods of time, your sympathetic nervous system is flawed into perpetual overdrive by lack of sleep. The consequential strain that is placed on your body by the persistent force of sympathetic activation will leak out in all manner of health issues, just like the failed pistons, gaskets, seals, and gears of an abuse car engine.

In the northern hemisphere, the switch to daylight savings time in March results in most people losing an hour of sleep opportunity. Researchers have discovered that this seemingly trivial sleep production comes with a frightening spike in heart attacks the following day. In the autumn within the northern hemisphere, when the clocks move back and we gain an hour of sleep opportunity time, rates of heart attacks plummet the day after. A similar rise and fall relationship can be seen with the number of traffic accidents.

Sleep loss and metabolism: diabetes and weight gain

In one study, participants were limited to sleeping four hours a night for just six nights. By the end of that week, the formerly healthy participants were 40% less effective at observing a standard dose of glucose, compared to when they were for the rest.

Inadequate sleep decreased concentrations of satiety signaling hormone leptin and increase levels of the hunger instigating hormone ghrelin

Recents study found that a lack of sleep makes your body produce endocannabinoids, chemicals very similar to the drug cannabis which stimulate appetite and increase your desire to snack.

Ample sleep can restore a system of impulse control within your brain, putting the appropriate brakes on potentially excessive eating.

When you’re not getting enough sleep, the body becomes specially stingy about giving up fat. Instead, muscle mass is depleted while fat is retained.

I report that studied over 40 years of more than 100,000 employed woman, working irregular nighttime hours (such as nurses who perform shift work) had a 33% higher rate of abnormal menstrual cycles than those working regular daytime hours.

Studies have found that short sleep duration will also disrupt activity of genes regulating cholesterol. In particular, a lack of sleep will cause a drop in high density lipoproteins, a directional profile profile that has consistently been linked to cardiovascular disease. Inappropriately time sleep, such as the imposed by jet lag or shift work, can I have equally large effects on the expression of human genes as inadequate sleep.

Neglect sleep, and you’re deciding to perform a genetic engineering manipulation on yourself each night, tempering with that nucleic// alphabet that spells out your daily health story. Permit the same in your children and teenagers and your imposing a similar genetic engineering experiment on them as well.

Part 3: How and Why We Dream

C9: Routinely Psychotic

Our emotional regions of the brain are up to 30% more active in REM sleep compared to when we are awake!

Since REM sleep is associated with an active, conscious experience of dreaming, it was perhaps predictable that REM sleep would involve a similarly enthusiastic pattern of increased brain activity.

This CEO region of your brain, which otherwise maintains your cognitive capacity for ordered, logical thought, is temporarily ousted each time you enter into the dreaming state of REM sleep.

REM sleep can therefore for be considered as a state characterized by strong activation in visual, motor, emotional and autobiographical memory regions of the brain, yet relative deactivation in regions that control rational thought.

Scientists were able to predict with significant accuracy the content of participants’ dreams at anyone moment in time using just the MRI scans, operating completely blind to the dream reports of the participants. Using the template data from the MRI images, they could tell if you were dreaming of a man or a woman, a dog or a bed, flowers or a knife.

My immediate reaction to this reasearch is fear. In the future this could mean that we could hold the dreamer responsible for what they dream. Is it fair to judge what it is they are dreaming, since they were not the conscious architect of their dream?

Part 4: From Sleeping Pills to Society Transformed

C15: Sleep and Society

Early studies demonstrated shorter sleep amounts, predict lower work, right and slow completion speed of basic tasks. Sleep deprived individuals also generate fewer and less accurate solutions to work relevant problems they are challenge with.

Studies also found that those individuals who obtained less sleep in the preceding days are the same people who consistently select less challenging problems. The opt for the easy way out, generating if you were creative solutions in the process.

The irony that employees miss is that when you are not getting enough sleep, you work less productively, and thus need to work longer to accomplish a goal. This means you often must work longer and later into the evening, arrive home later, go to bed later, and need to wake up earlier, creating a negative feedback loop.

Sleep and education

Keep in mind at 5:15 AM to a teenager is not the same as 5:15 AM to an adult. The circadian rhythm of teenagers shifts forward dramatically by 1 to 3 hours. So for a teenager to wake up at 5:15 AM in order to get to school by 7:20 AM is equivalent for adults to wake up at 3:15 AM day after day after day.

One school in Minnesota that moved start times from 7:25 AM to 8:30 AM not only gave 43 minutes of extra sleep to students but also improved their SAT academic performance.

Children from lower socioeconomic backgrounds are less likely to be taken to school in a car, in part, because their parents often have jobs in the service industry demanding work start times at or before 6 AM. Such children, therefore rely on the school buses for transit, and must wake up earlier than those taken to school by their parents. As a result, those already disadvantaged children become even more so because they routinely obtain less sleep than children from more affluent families. The upshot is a vicious cycle that perpetuates from one generation to the next, a closed loop system that is very difficult to break out of.

Sleep and Health Care

Residents working a 30 hour straight shift will commit 36% more serious medical errors, such as prescribing the wrong dose of a drug, or leaving a surgical implement inside of a patient.

One in 20 residents will kill a patient due to a lack of sleep.

I knew report has discovered that medical errors are the third leading cause of death among Americans after heart attacks and cancer. Sleeplessness, undoubtedly plays a role in those lives lost.

The Chernobyl nuclear power disaster was the fault of sleep-deprived operators, working an exhaustive shift, occurring, without coincidence, at 1 AM.

The Exxon Valdez oil tanker that spilled millions of gallons of crude oil and destroyed a coastal ecosystem happened because of a captain that had only slept six out of the previous 48 hours.

C16: A New Vision for Sleep in the Twenty-First Century

To convert a healthy new habit into a permanent way of life, you should have exposure to your own data. “Seeing is believing“ ensures longer-term adherence to healthy habits. This means using tools such as mobile apps to track your sleep and see how changes in your habit impact the your sleep patterns.

Public Policy and Societal Change

At the highest levels, we need better public campaigns educating the population about sleep. We spend a tiny fraction of our transportation safety budget warning people of the dangers of drowsy driving compared with the countless campaigns and awareness efforts regarding accidents, linked to drugs or alcohol. This despite the fact that drowsy driving is responsible for more accidents than either of these two issues – and this more deadly. The governments could save hundreds of thousands of lives each year if they mobilize such a campaign. It would easily pay for itself, based on the cost savings to the health care and emergency services bills that drowsy driving accidents impose. It would of course help lower healthcare and I’ll be insurance rates and premiums for individuals.

My Data: What can I learn from 4 years of sleep tracking?

For the past couple of years, I have used a mobile app on many (but not most) nights to track my sleep. By “track” I mean the app on my iPhone using the microphone to analyze noise around my movements in bed. I don’t trust this method entirely, but I do feel it is related somehow to how the way I ended up performing around the time my sleep improved or worsened.

Below I share the insights the app let me export I consider are the richest.

I was not paid in any way by the app developer nor is this an endorsement for you to implement this or with any other app. But I do recommend to investigate on current (and not-that-invasive) state-of-the-art ways for you to track your sleep and improve your health.

Now, onto the data!

Sleep Quality

“We calculate your sleep quality on a number of factors, but most importantly on the time you spend asleep, movements during the night and moments when you're fully awake. Use Sleep Cycle continuously, and it will learn and calibrate your quality score accordingly. We're all different even in sleep, which is why Sleep Cycle adapts its Sleep Quality algorithm to your patterns and sleep habits over time.”

Sleep quality

I can say right from the beginning what is going on here, specially in 2021. Up to that moment, my sleep worsened as I finished up my studies on Software Engineering, which took a big toll on my nerves, I must confess. Rushing to apply with the best marks as possible for 2018 and the first half of 2019, I made it to Vienna from Buenos Aires for a Double Degree program. The program was harder than expected, and I found myself pulling all-nighters frequently, but happily finished graduating in mid 2020.

The true minimum in quality was at the beginning of 2021, when I returned home for the Covid-19 pandemic, but kept the office hours of a timezone 4 hours later than Argentina. This meant waking up extra early for many weeks, but having a hard time going to bed earlier.

Then I returned to Vienna, later quit that job, and was much clearly more relaxed. I moved back to Argentina for good and my sleep quality has not fell from 60%, but at the time of writing it’s 20 points down from my all time high. Must be my addiction to Twitter before bed.

Location Factor

This is pretty insightful! I got my best sleep during vacations, and my worst during the hardest part of the Austrian lockdown.

Week times

Saturday a clear winner in terms of sleep. Friday the worse performer, but only because of the night outs 🕺!

Regularity

“Your Regularity is one of the most important factors for better sleep, but also a factor you can totally own. We all have habits that affect our well-being, and when you go to bed or rise in the morning is one of the strongest influencers. In fact, if you're sliding here, it's like being jetlagged without the travelling involved. Regularity tells you about your bedtimes and wake up times over a period of time. The higher it is, the more regular your bedtime and wakeup times, and that leads to better sleep. If you're not happy with this rating, just find your sleep beat and stick to it. Doesn't matter if you're a night owl or a lark. Results will come quickly either way.”

Regularity

Again, the double degree program (and no work at the time) took the worst out of my regularity. It took me the entirety of 2020 to recover from that; the lockdown did not help at all.

Regularity of the week

Same party-reasons here as before.

Went to Bed

“You'd be forgiven for thinking this one's pretty obvious. It measures when you go to bed. And that's it. Right? Not quite, because a key component to getting a good night's rest is to maintain your bedtime routines, and going to bed at about the same time every night, is a factor. Some argue that maintaining your bedtime routine is more important than getting enough hours of sleep, but until that's settled, we'll stick with just saying that it makes a difference. A potentially major one.”

Went to bed

Common denominator once again. Final thesis and lockdown made me feel I was living with Siberia’s timezone. A disaster! I remember waking up with the sunset of the Austrian winter.

It’s still bad. Provided my job starts at 9:30 am or later, I don’t see myself improving here.

Woke up

Remote work 4 hours behind clearly stands out here. Not a morning lark, am I?

Wake time during the week

Workdays are evident.

Efficiency

“Sleep efficiency is an important measure of the quality of your sleep. It represents the percentage of time spent asleep during the time you were in bed. For example, if you spent 8 hours in bed and slept for 7 hours, your sleep efficiency would be 87.5%. A high sleep efficiency score indicates that you are spending most of your time in bed asleep, which is important for feeling well-rested and alert during the day. On the other hand, a low sleep efficiency score may indicate that you are experiencing difficulties falling asleep or staying asleep throughout the night. Monitoring your sleep efficiency over time can help you identify patterns and make adjustments to your sleep habits that can improve your overall sleep quality. The Sleep Cycle app provides tools to track your sleep efficiency and offers insights and recommendations to help you optimize your sleep.”

Efficiency

This is different. It shows I am spending less time asleep in bed. Now I am at near all time lows, albeit a difference of ~5 points between min and max. I attribute this to late-night finance newsletter reading.

Asleep

“Time Asleep is where the real action is. This is where Sleep Cycle says you're not just in bed, but actually sleeping as well. While time isn't the sole factor to get a good night's sleep, it's so important that ignoring it, will influence everything, such as deep sleep, movements and REM sessions. If you don't clock the hours, you have less chance to recover, even if you're an expert sleeper. If you want to tweak just one thing, start here. Putting in the time to get the sleep you deserve, will help. We promise.”

Asleep

This is a shame, in historic standards. 5 hours is really bad If I consider the lessons of the book I just wrote about. I don’t feel bad though, and I have to say, I snooze the app a lot, so perhaps I get more sleep than recorded here. I would love to improve this.

Asleep after

Also a worse picture here. It takes me double the time to fall asleep than 3 years ago. I should really pay attention to iOS’s Bedtime reminder.

Farewell

Thank you for reaching the end! I hope this inspired you to improve your sleep, and to think how essential it is to our life, no matter your age or habits.

Let us never forget what it feels like to be truly awake during the day.

Examining the Bitcoin Whitepaper - Part 3/3

Sun, 03 Jul 2022 00:00:00 GMT

What’s this post about?

This is a three part post where I explain concepts included in the famous white-paper that are not introduced nor described beforehand, and are important to have a better grasp of the problems tackled by Bitcoin. They are divided into the same sections of the paper.

This is not a one-stop-shop for Bitcoin or crypto, but a good catch-most of the under-explained concepts in the original whitepaper. Also, this is not financial advise, nor am I trying to evangelize this to you. :)

How to make the best out of this post

I believe the best way to understand the original paper is to have this post by your side, stop here for the tricky and technological concepts, and return to reading the paper. There is no specific order, and you can skip any section (specially the mathy ones).

This is Part 3. Check Part 1 here, and Part 2 here.

Table of Contents
Combining and Splitting Value
- Fan-out
Privacy
- Unavoidable linking
Calculations
- What is a Binomial Random Walk?
- Why is q/p the probability the attacker catches up from 1 block behind?

Combining and Splitting Value

Fan-out

Why is there never the need to extract a complete standalone copy of a transaction’s history?

First of all, the BTC blockchain does not have a “balance” at any point.

If A sends a transaction to B, A creates an output for a certain amount of bitcoins. For A to spend those funds, an output script defines a condition to allow such event to occur: to spend the transaction, the spender (A in this case) must sign the transaction’s hash with the private key from A’s address. In this way, A can actually spend it, because A owns that unspent transaction. L

There are other possible scripts, not limited to 1 private key but multiple keys too, forcing other address to participate in the transaction.

So what is the balance I see on my wallet? Well, your wallet is actually showing you all unspent transaction outputs that you are able to spend with your private keys. These individual unspent outputs come from other transactions that are A’s right to spend. Let’s think of an example.

If you have 35 USD in your wallet, you must have a combination of bills for there is no 35 USD bill. For example, one 5 USD bill, one 20 USD bill and one 10 USD bill. Each of these individual bills are unspent outputs. They are values in your favour that you can spend and that you received from previous transactions that are in your wallet.

Let’s say now you want to spend 21 USD on a buffala pizza. You are going to give 1 or more of your unspent outputs to the cashier. So to cover the pizza, you have to give, say, the 20 USD bill and the 5 USD bill. The cashier then gives you 4 USD in change (say coins). This change is, yes, an unspent output, that goes back to your wallet with the rest of the outputs.

The key here is that the totality of the output must match the input. So in our example, you gave the cashier 25 USD, and then 25 USD in total were redistributed accordingly; 21 USD for the pizza place, and 4 USD back to you in change.

Note that, in the real world, if the total output amount is less than the total input amount, the difference is considered the transaction fee and this will go to the miner of the block your transaction will appear in.

If, on the other hand, the output is greater than the input, then the transaction is completely invalid. The difference would be created from thin air. You could give someone value it receive the same amount right back; the “double spend” Satoshi wants to avoid!

All in all, it is not possible to do a thing like “subtract the amount from person A’s balance and add it to person B’s balance”. When making transactions, you basically destroy some outputs to generate new ones.

So, if you didn't have to spend an output entirely, like in an account/balance system, you would need to check every other transaction that also includes that output and make sure the sum wasn't more than it contained, adding a lot of complexity to the validation and development.

Privacy

Unavoidable linking

Why is some linking still unavoidable with multi-input transactions, which necessarily reveal that their inputs were owned by the same owner?

In the transaction from above, 1LS1h8UJ… is both an input and a change address. Most of the transaction funds were sent to 187SRqCe…and the rest went back to 1LS1h8UJ… with 0.01580432 BTC.

This is a bad practice for privacy reasons. Someone watching can tell that the sender controls 1LS1h8UJ (used as input and change address).

If no address reuse was present then someone could not tell which was the destination address and which was the change address.

Always create new addresses for change.

Calculations

What is a Binomial Random Walk?

A random walk is sort of like the steps of a drunk person. One cannot figure out if the next step goes forward or backward (or left or right).

Formally, in a random walk, we start at some starting position, usually 0. Then subsequent steps are taken by looking at the current value and adding some random value to it.

In reality, steps look random, but are heavily dependent on previous values, namely the most recent one.

Why is q/p the probability the attacker catches up from 1 block behind?

Nakamoto isn’t analyzing whether the economics work out: it may be that the attacker spends more on mining than is recovered from the double spent transaction; or it may be that the Coinbase reward from announcing a tremendous number of new blocks is worth more than the double spent transaction. Nakamoto’s goal is instead solely to analyze the worst-case scenario where the attacker spares no expense in running their existing mining power to win the Gambler’s Ruin.

Let p be the mining power and probability that the honest miners find the next block, and let q be the attacker’s mining power. We define q + p = 1, since we assume either only the attacker or honest miners will win a block each round (and not some late comer or third-party).

If the attacker has unlimited resources for mining and stops when he reaches z, then we can use Eq. 22 from the paper quoted previously.

It’s worth noting an error by Nakamoto here. We aren’t interested in Qz, the probability that the attacker will simply catchup. Instead, Nakamoto should have calculated Qz+1, the probability of the attacker going one past the honest miners.

There are a few limitations we need to be aware of. First of all, it’s a challenge for the merchant to determine the attacker’s mining power q. It’s always possible that a previously unknown miner or an existing one could dedicate new resources to double spend attacks. The Fischer, Lynch, and Paterson (FLP) impossibility result tells us that Satoshi’s algorithm can never reach consensus. At any point in time, the last block on the chain is only an estimate of what consensus will be in the future.

Second, the model is a little strange in that the budget for the attacker is infinite for playing the Gambler’s Ruin; yet, the attacker has limited computational power. Satoshi was perhaps attempting to quantify the worst case for each value of q.

Examining the Bitcoin Whitepaper - Part 2/3

Sat, 02 Jul 2022 00:00:00 GMT

What’s this post about?

How to make the best out of this post

This is Part 2. Check Part 1 here, and Part 3 here.

Table of Contents
Network
- How does a node realize it missed one block?
Incentive
Reclaiming Disk Space
- Pruning
- How a UTXO Works
Simplified Payment Verification
- Has the strategy of network node alerts ever worked?

Network

How does a node realize it missed one block?

In the Bitcoin network, each peer maintains a full copy of the ledger (blockchain).

What would happen if a node lost one of in its blocks in the blockchain (e.g. another process on the peer accidentally wiped out the disk area that stores the block)?

Will this peer be able to detect that this block is missing? If yes, how will the peer detect it and how is it going to repair?

It depends on the implementation. For Bitcoin Core, the node maintains a database which has the locations of all blocks on disk.

If a block is deleted, then it will notice that the location of a block either does not exist or contains garbage or invalid data.

If it does notice that a block is missing (or something is corrupted), it will attempt to rebuild the database using the blocks stored on disk and then download the missing blocks from its peers.

However, blocks can actually be deleted after they have been verified and the local database has updated so that the node knows the block it has just deleted was a valid block.

This is called pruning, and with pruning, old, historical blocks can be deleted as they aren't used for anything in normal node operation (except helping new nodes sync).

Incentive

Because of the decreasing amount of BTC that is paid as a reward (due to the halving), the incentive for nodes to mine also decreases.

To compensate for that, Nakamoto introduces a secondary incentive: transaction fees.

These fees should replace, at some point in the future, the mined block reward.

At the beginning of 2022 block rewards still are greater than transaction fees, so miners still focus on them as their main source of income.

Either the fees need to increase significantly (a problem currently for Ethereum), or the price of BTC itself should climb much higher. These are the only ways in which the transition can occur.

If neither happens, miners will abandon the network, therefore lowering the security of the blockchain and entering a negative feedback loop.

Reclaiming Disk Space

Pruning

Pruning allows a node to free up storage space by removing transactions that are no longer needed to verify that the blockchain is in a consistent state (i.e. no bitcoins were created improperly).

Estimates of needing to store only a portion of data versus the ~500 GB we are approaching reduces the cost required at least for mobile storage by about $10.

As for growth, storage requirements for a pruned node scale more closely with how bitcoins are distributed in the unspent transaction set (a.k.a. UTXO). This means storage will grow O(the number of users) rather than with O(the number of transactions).

At these levels it starts to be practical for a phone or tablet to run a pruned node. This is useful for those that want a greater level of trust with their mobile wallets as well as those who wish to help Bitcoin nodes with bandwidth and connectivity challenges to keep up to date.

This development should help Bitcoin to extend its network reach and improve the trust level possible on commodity phones, tablets, and network equipment, be that mobile or otherwise.

How a UTXO Works

UTXO transactions sound complicated, but they really are fairly simple. UTXO or unspent transaction outputs are used in cryptocurrency transactions.

These are the transactions that are left unspent after someone completes a transaction, similar to the change someone receives after conducting a cash transaction at the store.

Here's how it works. A UTXO database is used to store change from cryptocurrency transactions. This database or ledger is initially set to empty or zero. As transactions occur, the database becomes populated with change records from various transactions.

When a transaction is completed and there are outputs that aren't spent, they are deposited back into a database as inputs that can be used at a later date for a new transaction.

Cryptocurrency transactions —such as those used for bitcoins— are similar to cashier checks. You cannot exchange them for custom amounts and must spend the entire amount stored in that data byte.

But cryptocurrencies like bitcoin are also unique in that transactions can be conducted using fractions of the cryptocurrency. This means spending does not take place using a single data byte.

Instead, multiple fractions of bitcoin are retrieved by the algorithm to fulfill a spending request.

For example, a purchase worth 1 bitcoin may retrieve 0.6 BTC from one byte and 0.4 BTC from another. Change from each of these fractions is then sent to the UTXO database to be spent at a later date.

Simplified Payment Verification

Has the strategy of network node alerts ever worked?

The network wide alert system was created by Satoshi Nakamoto as a means of informing Bitcoin users of any important information regarding Bitcoin.

It has been used in the past to inform users about important network events such as accidental blockchain forks.

However, the alert system also represents a large source of centralization in Bitcoin. The holders of the singular Alert Key can at any time send an alert which could affect the entire network.

As more developers join, the alert key is given to others, but cannot be taken away from those who have left.

This has led to the alert key potentially falling into the hands of malicious actors who could use it to disrupt the network.

Because there is only one alert key, it is not possible to prevent former developers from sending an alert nor is it possible to identify who sent an alert.

In addition, the alert system is primarily Bitcoin Core specific. Many other wallets have their own systems in place but still must have handling for the alert system because it is network wide. Something specific for one software should not be imposed on the entire network.

The alert system has also lost its usefulness. It is no longer necessary to use it to inform users about problematic network events as users can easily get their information from any major Bitcoin news outlet.

Examining the Bitcoin Whitepaper - Part 1/3

Fri, 01 Jul 2022 00:00:00 GMT

What’s this post about?

How to make the best out of this post

This is Part 1. Check Part 2 here, and Part 3 here.

Table of Contents
Introduction
P2P distributed timestamp server
- Cryptographic Hash
Transactions
- Mint
Proof of Work: Adam Back’s Hashcash
- Rewards for your Work
- Nonce

Introduction

The paper starts with an introduction on the trust inherent to the commerce on the Internet, particularly on the costs of transactions. At one point, Nakamoto talks about the impossibility of non-reversible transactions, and the cost of their mediation. Let’s stop here and dig deeper into what this means.

Trusted third parties like financial institutions, banks and credit card issuers are responsible for resolving disagreements between parties. In order to resolve them, they need to do extra checks which have a cost. Consequently, any transaction whose value is lower than the cost of validating it is simply not profitable and won’t be done.

So small transactions like buying a soda can or buying a digital copy of a song worth a couple of dollars are thus impossible, and the client is only left with more expensive options, like subscribing to a music service with a greater monthly cost.

Reversibility of transactions

But money is not the only cost in this. On the one hand, in order to be able to make complaints and reverse transactions, merchants/sellers first need information from their customers. On the other hand, if transactions were non-reversible they could be cleared without collecting any private information.

The non-reversibility of payments proposes two things:

merchants are protected since buyers can’t cancel their payments
buyers are protected through escrow mechanisms (delaying payment until the goods or services have been received)

Bitcoin transactions are irreversible and can only be refunded by the receiving party—a key difference from credit card transactions that can be canceled, through mechanisms which add costs to transactions and set a minimum viable transaction value.

The escrow mechanisms are not part of the solution proposed by the paper, but have been implemented by third parties, for a fee of course. Indeed, a middle man charging for escrow. Kind of contradictory, perhaps?

Escrow mechanisms

An escrow is a financial and legal agreement that protects both buyers and sellers in a transaction in exchange for a fee. This fee is paid to an independent third party that holds the payment until everyone carries out their responsibilities in the agreement.

As an example, if buyer B wants to buy something from seller S, B is at risk of being scammed by S if the latter does not deliver the goods or services of the transaction. How can B recover the BTC spent, given the fact that bitcoin is non-refundable?

Bitcoin escrow mechanisms exist for this purpose, most only accepting legal merchandise.

For illegal activities, which I hope the reader is far from being involved in, some BTC escrows, such as the Dark Bitcoin Escrow, accept illegal goods and services (stolen credit cards, drugs, unregistered guns, fake currency).

Naturally, these mechanisms are frequented by thieves and gang members who don't want to get scammed by other thieves and gang members.

Double Spending Problem

Later on, the authors propose a “solution to the double-spending problem using a peer-to-peer distributed timestamp server to generate computational proof of the chronological order of transactions.” There’s a lot to unpack here, so let’s go step by step.

Firstly, what’s the double-spending problem?

My favorite Reddit subreddit, r/explainlikeimfive, opens up possibilities for many wise surfers of the web to describe this sort of concept in a very didactic manner. The following explanation is a digested extraction of this and this.

Double spending is essentially the version of counterfeit money for digital money. With a digital currency like bitcoin, your money exists as a file. So what is stopping you from copying that file (money) and using both copies to buy things? It’s like buying a chocolate bar with one dollar bill, and a pen with another bill, but the bills are actually identical.

The blockchain comes to solve this. It is basically a history of all transactions of that digital currency. It works via agreement among all people who have copies of that blockchain. The key is that transactions on the blockchain are only considered valid if a majority of the people have agreed to that specific blockchain.

If you attempted to use a single unit of digital currency in two different transactions, it is possible that both transactions end up in different blockchains, but eventually one of them will be rejected and only the other one will be valid according to the majority of the network, so the one and only digital currency is "spent" and can't be used again.

As you may have noticed, the double spending problem in a digital currency system is fundamentally about the chronological order in which transactions happen. Let’s go over it once again with an example.

If A has $100 in their account, and tries to send $100 to each of B and C, obviously only B or C would get the money. Which of them gets it first depends on the order in which the transactions happen. If A sends $100 to B first, and then tries to send $100 to C, B would get the money and the second transaction would be rejected.

In a centralized digital currency systems (e.g. e-banking), there is a single entity (the bank) tracking transactions, so the double spending problem is “easy” to stop. Whichever transaction that entity receives first goes through and others are rejected. The chronological order of transactions (and thus which one is "first") is well-defined here since there is only a single entity responsible for tracking and ordering them.

However, in a decentralized digital current systems (e.g. Bitcoin), the order is not well-defined. If A broadcasts both transactions near-simultaneously, one person (or a validator node in the network of the blockchain) may receive the transaction to B first, another person may receive the transaction to C first.

Thus, Bitcoin ends up being the first successful system to solve the double spending problem with blockchain technology.

The blockchain allows a decentralized crowd to record and order information (in this case, transaction data) that is extremely difficult to modify once recorded. This allows all Bitcoin nodes to agree as to which of the two conflicting transaction happened "first".

Secondly, what’s a peer-to-peer network?

Peer-to-peer Network

In a P2P network, every participant plays the same role; they are all peers. Each peer is both a client - requesting data, and a server - providing data. When you compare it to a regular website, this makes it a lot more robust.

A peer is usually connected to several other peers and if one goes offline, everything still works as usual. This makes blockchain networks very robust.

The nature of the network being P2P allows it to avoid any third party, any financial institution, and instead depend on the effort of each and everyone of the peers involved.

Thirdly, what is a timestamp server?

P2P distributed timestamp server

In Bitcoin’s P2P network, all the transactions are public to all the participants of the networks, namely, the nodes.

This public record attempts to solve the double spending problem. But enforcing all nodes to agree at all times about the single source of truth of the transactions is not a trivial task.

The solution proposed includes making the nodes agree on the order of transactions. This means that if two transactions involve the same coin (double spending it), then the one that is kept will be solved.

So what is the criteria used to decide which one comes first? Nakamoto proposes a timestamp server.

A timestamp server works by taking a hash of a block of items to be timestamped and widely publishing the hash, such as in a newspaper or Usenet post [2–5]. The timestamp proves that the data must have existed at the time, obviously, in order to get into the hash. Each timestamp includes the previous timestamp in its hash, forming a chain, with each additional timestamp reinforcing the ones before it.

Firstly, a timestamp is just a string of characters that identify a specified date and time of day, often with seconds to add to the accuracy and a timezone indication. For example, “20/12/2021 12:00:00 -0400”.

Generally, a timestamp is added to something else. In this case, a collection of transactions, a block, will be timestamped. The transactions in it don’t compete with each other to be the last one standing in the chain, so grouping them together is more efficient than having each transaction hold one timestamp. The timezone of reference is London local time, UTC-0.]

Secondly, the block of transaction or items is then “hashed” and made public to the rest of the network. But what is a hash?

Cryptographic Hash

In plain terms, a hash is a function that receives an input A (of arbitrary length, and mostly an alphanumeric text) and returns an output B (often of a specific size, like 256 characters) in such a way that it is extremely hard to create an inverse function, that is, a function that takes B and outputs A.

A ‘good’ hash:

Is easy to calculate for any input.
Makes it extremely computationally difficult to calculate or find an input that has a given hash.
Has an extremely low probability of having two different messages with the same hash.

So to put it shortly, a hash is a short string of charaters that can be computed easily, but cannot be reversed. It’s a small pseudo-random summary or fingerprint of the input. This fingerprint determines the exact moment in which the block of transactions was mined and validated.

In relation to the timestamp, the hash is made from it, among others things (previous block header hash, difficulty, nonce and Merkle root). The moment the hash is published, the block’s data (including the timestamp that the hash sums up) guarantees that it must have existed at the time of creation.

Thirdly, but most importantly, as new blocks of transactions are put together, each new hash is calculated with the previous block’s hash. If you trace the path from the latest block in the blockchain, jumping blocks one hash at a time backwards, you will eventually reach the genesis block, the very first block that was produced.

I will spare you the search of when this block’s timestamp: January 3rd 2009, 18:15:05h UTC.

This traceability guarantees a linear order of blocks in which the entire network agrees upon. To mess with this chain, an attacker would have to brake the hash of the target block, and every single block after it; a practically impossible task (at least as of the time of writing). So if a block B’s hash was computed from a block A’s hash, this proves that the block A came before block B.

As hard as it is to rewrite the chain’s history, it is computationally easy to verify a block’s hash: anyone can check that block B’s hash results from block A’s hash by recomputing it. This verification is key for network nodes to rest assured that any incoming blocks (and their transactions) have not been modified in any way.

Transactions

By introducing the chain of digital signatures, Nakamoto states that the payee can verify the signatures to verify the chain of ownership. But a problem is then posed. How can a payee verify that the received coin was not double-spent?

A common solution is to introduce a trusted central authority, or mint.

Mint

Investopedia defines mint concisely:

A mint is a primary producer of a country's coin currency, and it has the consent of the government to manufacture coins to be used as legal tender.

Along with production, the mint is also responsible for the distribution of the currency, protection of the mint's assets, and overseeing its various production facilities.

Proof of Work: Adam Back’s Hashcash

Now that we saw what a P2P distributed timestamp server is and how blocks of transactions are hashed, let's dive deep into how this all works together.

Hashcash uses a proof-of-work technique to prove an email is not spam. Bitcoin mining also uses a proof-of-work system, but in its case, to prove a block of transactions is valid. In order to implement a distributed timestamp server on a peer-to-peer basis, Satoshi mentions the need of a mechanism that validates transactions on the network.

The idea behind Hashcash is pretty simple. In order to prove an email is not spam, the sending software must solve a small but CPU-intensive math problem. The answer to the math problem is included in the email header, which is read by the receiving email software.

This prevents spam because spammers send millions of emails each day. Having to solve a small (but CPU-intensive, aka, high electricity costs) math problem for each of those millions of emails would require a lot of very expensive hardware, which makes sending spam /unprofitable/.

Rewards for your Work

The whole point of Bitcoin mining is verifying transactions. Mining doesn't solely exist to generate more coins. The coins that are generated are given to the miners as a reward for verifying transactions. People will still have to mine even when there are no more coins to generate, because transactions still need to be verified forever and ever.

A transaction occurs anytime someone sends Bitcoins to another person. Each transaction needs to be verified to ensure it's legit, e.g. ensuring the sender did in fact send the coins, and ensuring the sender had the coins to send.

Without verifying transactions, people could create their own fake coins and send them to themselves, or they could pull a double spending attack, which, as we have seen, is sending the same coins to more than one person.

So someone (or something) needs to verify each transaction. But Bitcoin is a distributed system without any central authority, so who/what exactly verifies transactions?

The answer is the miner who did the most work to prove the transaction. This is where proof-of-work comes into play. In order for a block of transactions to be verified and added to the blockchain, a miner must solve a very complicated math problem. A problem that is so complicated that miners need to buy restrictively expensive hardware, and it's impossible to predict which miner will solve the problem.

The verification process prevents double spending because the person trying to execute the double spend would have to verify their own fake transaction, which, because of proof-of-work, is impossible.

It's impossible to verify your own transactions because of the complicated random nature of the math problem that needs to be solved. There are millions of other miners trying to verify the same transactions, and the chances of your miner verifying the transaction is practically zero. You are competing with /every other miner on the planet/ to find the next block. If you "win", you get the block reward plus you get to choose which transactions are included in your block. If somebody else wins, you get no say in the matter. Unless you have a very large mining operation, the chance of you winning is so small that it's never going to happen.

So miners verify transactions by solving complicated math problems, and as a reward the Bitcoin network "rewards" the miner who verified a block of transactions with a set number of coins.

In fact, the miner who solved the problem creates the coins and gives them to themselves!

When no more Bitcoins can be generated, miners will continue to mine to collect the fee attached to each transaction, paid by the sender.

Nonce

Nonce, or a "number only used once," refers to the first number a blockchain miner needs to discover before solving for a block in the blockchain.

Adding transactions to the blockchain requires substantial computer processing power. The individuals and companies who process blocks, the miners, are compensated only if they are the first to create a hash that meets a certain set of requirements, called the target hash.

The process of guessing the hash starts in the block header. It contains the block version number, a timestamp, the hash used in the previous block, the hash of the Merkle Root (a data structure with references to other hashes), the nonce, and the target hash.

If the hash meets the requirements set forth in the target, then the block is added to the blockchain. Cycling through solutions in order to guess the nonce is the famous proof of work, and the miner who is able to find the value is awarded the block and paid in the cryptocurrency.