This thesis examines topics in ultrametric physics, by which is meant physical theories involving $p$-adic numbers or the $p$-adic distance function. Such studies on the one hand provide a toy-model for understanding string theory and AdS/CFT in a simple setting that we will leverage to carry out otherwise unmanageable computations, and on the other hand novel kind of theories framed in terms of real and $p$-adic numbers admit of applications to exotic condensed matter systems. Chapter 3 is based on work with S. Gubser, Z. Ji, and B. Trundy. We present a version of the $O(N)$ model defined over a space with real and $p$-adic directions that is relevant to spin chains with hierarchical coupling patterns that are experimentally realizable in cold atom labs. For this model, we establish the existence of a Wilson-Fisher fixed point and compute anomalous dimensions around this point. Chapter 4 is based on work with S. Gubser, Z. Ji, B. Trundy, and A. Yarom. We discuss how to couple $p$-adic string theory to a curved target space, and we formulate a related scalar theory described by an action given by a bi-local integral of a power law factor, with an exponent parameter $s$, multiplied by the squared arc length between points on the target manifold. Depending on the choice of $s$ and of dimension, the theory can reduce to the sigma model, higher derivative versions thereof, or to a non-local theory with possible applications to the dynamics of membranes. We study the renormalization of the theory at one-loop level. Chapter 5 is based on work with S. Parikh. We develop a Mellin space formalism for $p$-adic AdS/CFT and present rules for evaluating Mellin amplitudes at tree-level. We analyse the structure of these amplitudes and the features shared with Mellin amplitudes in real AdS/CFT, including a Mellin-Barnes integral representation wherein Mellin amplitudes, whether real or $p$-adic, assume the same form. Chapter 6 is based on work with S. Parikh. Inspired by formulas from $p$-adic AdS/CFT and a number of simple computations that apply these formulas, we derive a set of propagator identities in the context of real AdS/CFT and employ them to determine the six-point conformal block in the snowflake channel and perform the conformal block decomposition of five- and six-point AdS diagrams.