IPS Meeting 2024 and Nobel Week

Last week I attended the IPS Meeting 2024, held this year at Nanyang Technological University, and gave a plenary talk on flatband lattices, covering material we recently published in an invited review in Nanophotonics. Among the many interesting talks this year, the plenary presentation by Antonio Castro Neto on the creation of carbon thin films and liquid crystals via oxidation of graphene (see e.g. this paper).

This week the Nobel Prizes were announced, with machine learning and AI dominating the Physics and Chemistry prizes. It's quite remarkable that one of the Physics laureates (Hopfield) published his prize-winning work as a single author theory paper in 1982, when he was already 49 years old! It's never too late to do your most impactful work!

From large language models to local language models

Last week Nature published a feature on local AI: Forget ChatGPT: why researchers now run small AIs on their laptops

This article discusses developments in large language models (LLMs) leading to the proliferation of language models that can be run locally on your own device without requiring top of the line hardware. There are four driving motivations behind this:

Privacy: Cloud-based LLMs such as ChatGPT do not offer any user privacy. This is a no-go when wanting to use them to analyze any kind of proprietary or confidential data. The only way to guarantee privacy is if you have a model that doesn't need to communicate with some cloud server to run.

Reliability: LLMs are constantly evolving. With commercial providers, there is a tug-of-war between the providers and the users, many of whom explore methods to "jailbreak" a model using finely crafted inputs to escape hard-coded restrictions on the possible outputs. Even when the underlying LLM might stay the same, preprocessing applied to a user's input before querying the LLM might change as the provider aims to improve the model performance or accuracy. This makes LLMs inherently unreliable - a prompt that works today might fail hopelessly the next day. With a local LLM the user is in control and will not be surprised by sudden changes to the model performance. Note that running a LLM locally does not completely solve this issue, since there is always some randomness to their output.

Reconfigurability: With the advent of efficient LLM fine-tuning methods such as low rank adaptation (LoRA), users can take an off-the-shelf open source LLM and augment it with their own specialized or proprietary data to solve problems of interest. For example, for the first year maths course I'm currently teaching the course convenor has augmented a LLM with the lecture notes and problem sets, creating a chatbot that is able to answer students' questions about the course and also refer them to the relevant parts of the lecture points. For the students, this combines the ease of use provided by a chatbot with the reliability of the source materials.

Cost: For heavy users cloud-based LLMs are not cheap. Moreover, academics need to make the decision between paying for access out of their own pocket, or wading through their institution's bureaucracy to find some funding source that will cover a subscription. Local LLMs avoid these hassles.

The feature article also lists popular platforms for installing and using local LLMs, both command line-based (for power users) and GUI-based (for ease of use). As a backend, many of these packages rely on fast execution of LLMs provided by llama.cpp, which I covered previously here and here.

It's been a while since I tinkered with these packages, but clearly there have been quite significant developments in their performance and usability since I last used them more than a year ago!

Postdoctoral Position at Wave Transport in Complex Systems Lab—Wesleyan University

The Wave Transport in Complex Systems (WTICS) Lab at Wesleyan University is
opening a post-doctoral position on wave transport in theory or/and experiment
using microwave and RF analogue circuitry. The candidate must have a basic
knowledge of theory of metamaterials. A knowledge of software packages for
electronic design (such as COMSOL, SPICE etc) is desirable. Specific areas of research that are relevant to the current post include:

(a) Artificial Intelligence and Machine Learning;
(b) Non-Hermitian systems;
(c) Active topological structures;
(d) Asymmetric transport;
(e) Wavefront shaping techniques

The position is for two years with a possible extension of a third year. Information
about the WTICS group can be found here. Interested candidates should send a CV, a short statement of work and three recommendation letters to Prof. T. Kottos at tkottos@wesleyan.edu.

Postdoctoral Positions at Nankai University, Tianjin, China

Applications are solicited for postdoctoral positions in experimental/theoretical optics and photonics in the research group of Prof. Zhigang Chen/Hrvoje Buljan at Nankai University, China, which is one hour away from the capital Beijing and the alma mater of Shiing-Shen Chern. The areas of emphasis are topological photonics, nonlinear optics, optical trapping and manipulation, and machine-learning photonics. A PhD in physics, optics, or related area is required. It is expected that the candidate should have basic numerical skills and/or optical experimental skills and research experience with a good track of record in publications.

The initial appointment will be for three years, with the possibility of extension pending on performance and research funding. If accepted, the salary is competitive (annual gross salary > RMB 500,000, about $70,000), with possible additional merit award depending on academic performance evaluated at the end of each year. Free on-campus apartment is available, and the living cost is very low compared to income. The position will be funded under the China Postdoctoral International Exchange and Introduction Program, with the purpose of attracting outstanding PhD graduates to join the university and conduct postdoctoral research.

Applicant eligibility:
1. In general, applicants shall be under the age of 35. As for key disciplines supported by the university, the limitation on age can be eased to 38 years old.
2. Both Chinese and foreign graduates obtaining their PhD (from overseas top universities or supervised by an internationally recognized scientist) in recent 3~5 years can apply.
3. PhD candidates who meet the above conditions can also apply if they can start the postdoctoral position by June 2025.

Interested candidates are encouraged to contact Professor Chen or Professor Buljan before April 20, 2024 for application this year. However, the starting date is flexible.

Website: https://topo-photonics.nankai.edu.cn/index.htm

Transformer quantum states: more than meets the eye?

The large language models that have boomed in power and popularity over the last year are based on a neural network architecture called the transformer. Transformers were originally designed to efficiently learn complicated long range correlations arising in natural language processing and are now being applied to other areas, including many-body quantum physics, where they being used as flexible variational quantum states. 

Interest in neural network quantum states has grown rapidly since 2017, when Carleo and Troyer showed that a neural network architecture called the Restricted Boltzmann Machine could be trained to find the ground state wavefunction of the transverse-field Ising and antiferromagnetic Heisenberg models. One limitation with this original work was the difficulty of computing expectation values of the ground state using this architecture, since the trained network takes as its input a spin configuration and returns the corresponding probability amplitude, meaning that time-consuming Monte-Carlo sampling is needed to evaluate expectation values.

Monte-Carlo sampling can be avoided using different parameterizations of the many-body quantum state. For example, the autoregressive quantum state encodes the many-body wavefunction $\Psi(\boldsymbol{s})$ as a product of conditional probability distributions:

$$\Psi(\boldsymbol{s}) = \prod_{i=1}^N \psi_i(s_1 | s_1,...,s_{i-1}) = \psi_1(s_1) \psi_2(s_2|s_1) \psi_3 (s_3 | s_2 s_1) ...\psi_N(s_N| s_{N-1}...s_2 s_1) $$

One can thereby draw unbiased samples from the many-body ground state by first drawing the first spin, $s_1$, according to the learned probability distribution $\psi_1(s_1)$, followed by $s_2$ according to the conditional probability distribution $\psi_2(s_2 | s_1)$, and so on until a complete spin configuration is obtained. However, there is a conservation of misery in that we avoid Monte Carlo sampling but instead need to learn an exponential number of conditional probability distributions! Luckily, it is empirically observed that a single neural network is able to encode all this information, if we allow it to take as an additional input a hidden vector $h_i$ that encodes information as to the previously-drawn spins. This gives the neural network an autoregressive structure, in that the output from one pass is sequentially fed back into its input.

The autoregressive quantum states were inspired by the autoregressive neural networks developed for natural language processing tasks. The performance and scalability of autoregressive neural networks is limited by the need for sequential processing to generate a single sample, the potential for vanishing gradients making it difficult to train the network, and a bias of the learned probability distribution to the recently-sampled spins. More advanced formulations based on masked convolutional networks are able to generate entire configurations using a single evaluation of the network, but still encounter issues with trainability and encoding distributions exhibiting complex correlations.

Then along came the transformer architecture. The innovation here was the inclusion of multiple parameterized transformations to the input data that can be trained to pick out different features and (long-range) correlations – the multi-head attention. Once the key features are identified by the multi-head attention, a relatively simple feed-forward neural network is sufficient to compute the output probability. The success at this architecture for language modelling is now inspiring many studies on applications to many-body physics.

One line of research is exploring transformer neural networks as a flexible ansatz capable of describing families of many-body quantum systems, exemplified by the paper "Transformer quantum state: A multipurpose model for quantum many-body problems." In this work, the transformer neural network was trained to learn the many-body ground states of a family of Ising models. Thus, it takes as its input model parameters (e.g. the applied magnetic field strength), and then draws samples of ground state spin configurations. The model can also extrapolate to predict properties of ground states not included in the training data, albeit with lower accuracy, particularly when attempting to extrapolate across a phase transition. The network can also be "inverted" to perform a maximum likelihood estimation of the system's parameters given a few spin configurations drawn from its ground state, analogous to shadow tomography of many-body quantum states.

A second line of research is exploring the use of transformers as a means of accurately computing ground state energies from specific strongly-correlated model Hamiltonians, such as the Shastry-Sutherland model, see for example the paper "Transformer Variational Wave Functions for Frustrated Quantum Spin Systems". In this case, an architecture called the vision transformer is trained to learn the complex correlations present in the ground state. The biggest challenge is training the network, which is particularly difficult for complex-valued networks, however recent work has shown that a two-stage architecture that applies a real-valued transformer followed by a complex fully-connected neural network can be trained more easily.

What next for this hot topic? Better training methods or more easily-trainable transformer architectures are needed, since in training data for quantum many-body systems is much harder to come by than web-scraped training data for large language models. Future research on applications of classical transformer neural networks will likely be divided between problem-specific models tailored to solve certain hard many-body problems, and less accurate general purpose "foundation" models which may be useful for generating initial guesses for other more precise iterative methods. Beyond this, quantum and quantum-inspired generalisations of the transformer architecture are also cool directions to watch!

A busy January

There's been a lot going on here...

Machine Learning & Physics

Unsupervised learning of quantum many-body scars using intrinsic dimension - Now available on arXiv! We applied manifold learning techniques to identify scar states in the PXP model The take-home message: manifold learning techniques are a powerful alternative to more popular deep learning methods, especially in physics problems where you might not have access to enough training data for deep learning to work well.

Identifying topology of leaky photonic lattices with machine learning - Just published in Nanophotonics! We apply various machine learning methods to distinguish different topological phases in a photonic lattice, assuming one only has access to intensity measurements. This can serve as an alternative to full state tomography or phase retrieval methods, but one needs to be careful when training the models on ideal / pristine systems and then applying them to disordered systems. The journal also published a press release on WeChat!

Quantum Computing

Computing electronic correlation energies using linear depth quantum circuits - Finally published in Quantum Science & Technology, after more than a year and a half working through the peer review system. We use perturbation theory to determine electronic correlation energies in small molecular systems (hydrogen, lithium hydride, etc.) using a large set of shallow circuits, giving an alternative to existing methods which require deeper circuits infeasible for current quantum processors. We also tested the algorithm on cloud quantum processors, observing the detrimental impacts of noise. It would be interesting to run this again now to see how much (or how little) the performance from the different cloud providers has improved!

Landscape approximation of low-energy solutions to binary optimization problems - Published in Physical Review A. We present a method to obtain approximate solutions to binary optimization problems using the localization landscape, a function which is able to place bounds on the regions of Anderson localized eigenstates in disordered media without solving the underlying eigenvalue problem. We lay out the conditions required for these bounds to hold, outline how a quadratic unconstrained binary optimization problem can be transformed to fit these conditions, and provide details on how the quantum state representing the landscape function can be produced and sampled using techniques developed for near-term quantum devices.

 

On a related note, I was interested to see this month a new arXiv preprint in which the localization landscape was used to engineer multifractal resonances in SiN membranes!

Photonic Flatband Resonances

Photonic Flatband Resonances in Multiple Light Scattering - Published in Physical Review Letters. We reveal that flatbands can emerge as collective resonances in fine-tuned arrays of Mie-resonant nanoparticles, leading to giant values of the Purcell factor for dipolar emitters. The article was also highlighted with a Synopsis in Physics Magazine!

Physics meets machine learning and AI

Machine learning research of interest to physicists can be broadly divided into two categories: using machine learning tools to solve physics problems, and using ideas from physics to improve machine learning techniques.

An example of the former is the transformer neural networks used in the design of large language models such as ChatGPT. The ability of the transformer neural network architecture to efficiently learn long-ranged correlations in data is also useful for variational methods for finding ground states of strongly-correlated quantum many-body systems. Two papers demonstrating this approach were published in Physical Review B and Physical Review Letters earlier this year.

Popular image generation tools such as Dall-E and Stable Diffusion (which I wrote about previously) are based on time-reversing a diffusion process to generate desired samples from noise. This approach is heavily inspired by techniques from non-equilibrium statistical mechanics published in Physical Review E in 1997.

Another pressing issue in machine learning and AI is how to understand the emergent properties of large language models as their size or training time is scaled up. This is a problem that physicists are well-posed to tackle using techniques from statistical physics, random matrix theory, and the theory of phase transitions, which have recently been applied to shallow neural network models in a few different studies:

Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models

Learning through atypical phase transitions in overparameterized neural networks

Grokking phase transitions in learning local rules with gradient descent

Droplets of Good Representations: Grokking as a First Order Phase Transition in Two Layer Networks

I'm sure we'll see a growing number of theoretical physicists becoming involved in this exciting area of research in the coming years.

arXiv highlights

Quantum-noise-limited optical neural networks operating at a few quanta per activation

Suitably-trained optical neural networks can still perform classification tasks accurately using low intensity light with a low signal to noise ratio. This suggests that specialized light-based analogue hardware for machine learning may offer a route towards reducing the enormous energy consumption of neural networks!

Dissipative mean-field theory of IBM utility experiment

Another approach towards reproducing the results of IBM's kicked Ising model quantum simulation experiment, this time using mean field theory. The Appendix gives a simple rule of thumb for estimating the quantum volume of specific devices based on their two-qubit gate and readout fidelities and compares some different hardware providers.

Maximally-Localized Exciton Wannier Functions for Solids

Wannier functions - localized states constructed as a superposition of Bloch waves from an energy band of interest - are an important tool of the condensed matter physicists' trade. This work presents a method for constructing maximally-localized Wannier functions for multi-particle states, focusing on applications to excitons (electron-hole pairs).

Tensorized orbitals for computational chemistry

This work presents a tensor network-based compression of the matrix elements that need to be computed and stored when performing quantum chemistry calculations, based on Tensor Cross Interpolation. This is yet another example of how tools from quantum many-body physics can be used to speed up time-consuming computational tasks - no working quantum computer needed!

Recommendation systems for papers

The volume of papers being published and preprints being submitted to the arXiv has grown enormously since the time when I started my PhD studies. Moreover, competing preprint platforms such as Optica Open have been launched. This deluge of papers is impossible to keep up with. 

Therefore, there is growing interest in developing advanced bibliometric tools in order to stay abreast of the most important developments in one's own research field. There are two distinct approaches to solving this issue. 

The first is crowdsourcing. Websites such as scirate (founded by and most widely used by quantum physicists) and PubPeer (seems to be popular in life sciences) and allow users to upvote and comment on preprints they think are interesting. Preprints that are upvoted more appear higher on the page and therefore receive more attention and more views. The idea then is that the most important works will be upvoted more and will be seen by more people.

Other approaches are based on machine learning or bibliometric analysis methods where a new paper is analyzed by model that takes as its input various attributes of the paper, such as the authors, the topic, keywords, and its reference list. Models can be trained to pick out papers which are likely to relevant or more important and show them to the end user. 

One example of this now supported by arXiv is Litmaps, which constructs a graph that places an article in the context of previous and subsequent works. The visualisation can be customised to highlight different features, such as the publication date and total number of citations in the plot below. It seems at least that for the example below the "Seed" map seems biased towards highlighting review articles (missing hot recent results such as those on quantized Thouless pumping of solitons). The other visualisations offered are "Discover" (for finding overlooked papers) and "Map" (for telling a "research story"), but they require an account, and presumably a subscription for serious use.

Litmap of "Edge solitons in nonlinear photonic topological insulators"

Each approach has its own pros and cons. One problem with the crowdsourcing approach is that it can be sensitive to initial perturbations and tends to amplify existing well-known authors while inhibiting the promotion of less well-known authors. For example, people may upvote a paper just because it is written by familiar names and the title and abstract look interesting, leading to a winner-takes-all effect. 

This I think is a particularly important problem in research. At least my style is that I prefer not to work on ideas that are already very popular. I think to really make a deep breakthrough in research we need to see something that nobody else has noticed before. This often will involve getting insights from papers which have been forgotten or overlooked by the wider community. In this context crowdsourcing has the danger of leading to a groupthink in which the popularity of certain topics may exceed their promise, due to people working on them simply because many other people are also working on them. Thus, the choices of a few early adopters or "academic influencers" will end up getting amplified more and more.

The machine learning approach has the potential to give a more thorough and systematic coverage of the literature by being able to analyze all new papers and find the most interesting ones without being subject to this reliance on or sensitivity to initial fluctuations and early upwards. While there is promise of course, the big question is how are you supposed to develop a model to rank individual papers without studying deeply the science they contain? And how much can you trust a proprietary, closed-source model whose inner workings and potential biases are unknown?

For example, a crude first approximation might involve analyzing the references of a new paper to see what is cited. If a new paper cites important previous results (which can be estimated by how often they have been cited) then hopefully the paper will be worth reading. However simply counting the raw citations or references will be prone to bias. Different fields have different standards as to what is and should be cited. In some fields now you will see paper introductions which cite dozens or even hundreds of papers. In this case the value of an individual citation is relatively low and so looking at the citations alone we won't give much information as to what the paper is roughly about or whether it is worth reading.

It seems therefore that more sophisticated approaches are necessary. The context of a citation is important. Papers cited in bulk are not that valuable. For example, in an introductory paragraph "x has received a lot of interest lately [1-103]" only reveals that x is a hot topic. On the other hand, a sentence like "We apply the method of Ref. [16]..." tell us that there is probably a very close connection to whatever Ref. [16] is about. Thus, the integration of large language models such as BioGPT with paper recommendation systems is likely to improve their performance, thereby greatly improving the productivity of researchers who use them.

What do you think? Are there any other tricks for keeping up with the literature in your field?