publications

2021

1. Backretrieval: An Image-Pivoted Evaluation Metric for Cross-Lingual Text Representations Without Parallel Corpora Fain, Mikhail, Twomey, Niall, and Bollegala, Danushka arXiv preprint arXiv:2105.04971 2021 [Abs] [PDF]

   Cross-lingual text representations have gained popularity lately and act as the backbone of many tasks such as unsupervised machine translation and cross-lingual information retrieval, to name a few. However, evaluation of such representations is difficult in the domains beyond standard benchmarks due to the necessity of obtaining domain-specific parallel language data across different pairs of languages. In this paper, we propose an automatic metric for evaluating the quality of cross-lingual textual representations using images as a proxy in a paired image-text evaluation dataset. Experimentally, Backretrieval is shown to highly correlate with ground truth metrics on annotated datasets, and our analysis shows statistically significant improvements over baselines. Our experiments conclude with a case study on a recipe dataset without parallel cross-lingual data. We illustrate how to judge cross-lingual embedding quality with Backretrieval, and validate the outcome with a small human study.

2. Multi-Field Models in Neural Recipe Ranking – An Early Exploratory Study Takiguchi, Kentaro, Twomey, Niall, and Vaquero, Luis M arXiv preprint arxiv:2105.05710 2021 [Abs] [PDF]

   Explicitly modelling field interactions and correlations in complex documents structures has recently gained popularity in neural document embedding and retrieval tasks. Although this requires the specification of bespoke task-dependent models, encouraging empirical results are beginning to emerge. We present the first in-depth analyses of non-linear multi-field interaction (NL-MFI) ranking in the cooking domain in this work. Our results show that field-weighted factorisation machines models provide a statistically significant improvement over baselines in recipe retrieval tasks. Additionally, we show that sparsely capturing subsets of field interactions offers advantages over exhaustive alternatives. Although field-interaction aware models are more elaborate from an architectural basis, they are often more data-efficient in optimisation and are better suited for explainability due to mirrored document and model factorisation.

3. Statistical Hypothesis Testing for Class-Conditional Label Noise Poyiadzi, Rafael, Yang, Weisong, Twomey, Niall, and Santos-Rodrı́guez, Raúl arXiv preprint arxiv:2103.02630 2021 [Abs] [PDF]

   In this work we aim to provide machine learning practitioners with tools to answer the question: is there class-conditional flipping noise in my labels? In particular, we present hypothesis tests to reliably check whether a given dataset of instance-label pairs has been corrupted with class-conditional label noise. While previous works explore the direct estimation of the noise rates, this is known to be hard in practice and does not offer a real understanding of how trustworthy the estimates are. These methods typically require anchor points - examples whose true posterior is either 0 or 1. Differently, in this paper we assume we have access to a set of anchor points whose true posterior is approximately 1/2. The proposed hypothesis tests are built upon the asymptotic properties of Maximum Likelihood Estimators for Logistic Regression models and accurately distinguish the presence of class-conditional noise from uniform noise. We establish the main properties of the tests, including a theoretical and empirical analysis of the dependence of the power on the test on the training sample size, the number of anchor points, the difference of the noise rates and the use of realistic relaxed anchors.

2020

1. Towards Multi-Language Recipe Personalisation and Recommendation Twomey, Niall, Fain, Mikhail, Ponikar, Andrey, and Sarraf, Nadine In In Fourteenth ACM Conference on Recommender Systems (RecSys ’20) 2020 [Abs] [PDF]

   Multi-language recipe personalisation and recommendation is an under-explored field of information retrieval in academic and production systems. The existing gaps in our current understanding are numerous, even on fundamental questions such as whether consistent and high-quality recipe recommendation can be delivered across languages. In this paper, we strive to answer these questions by introducing the multi-language recipe recommendation setting and presenting grounding results that will help to establish the potential and absolute value of future work in this area. Our work draws on several billion events from millions of recipes and users from Arabic, English, Indonesian, Russian, and Spanish. We represent recipes using a combination of normalised ingredients, standardised skills and image embeddings obtained without human intervention. In modelling, we take a classical approach based on optimising an embedded bi-linear user-item metric space towards the interactions that most strongly elicit cooking intent. For users without interaction histories, a bespoke content-based cold-start model that predicts context and recipe affinity is introduced. We show that our approach to personalisation is stable and easily scales to new languages. A robust cross-validation campaign is employed and consistently rejects baseline models and representations, strongly favouring those we propose. Our results are presented in a language-oriented (as opposed to model-oriented) fashion to emphasise the language-based goals of this work. We believe that this is the first large-scale work that comprehensively considers the value and potential of multi-language recipe recommendation and personalisation as well as delivering scalable and reliable models.

2. Neural ODEs with stochastic vector field mixtures Twomey, Niall, Kozłowski, Michał, and Santos-Rodrı́guez, Raúl In Proceedings of the Twenty-Fourth European Conference on Artificial Intelligence (ECAI ’20) 2020 2020 [Abs] [PDF]

   It was recently shown that neural ordinary differential equation models cannot solve fundamental and seemingly straightforward tasks even with high-capacity vector field representations. This paper introduces two other fundamental tasks to the set that baseline methods cannot solve, and proposes mixtures of stochastic vector fields as a model class that is capable of solving these essential problems. Dynamic vector field selection is of critical importance for our model, and our approach is to propagate component uncertainty over the integration interval with a technique based on forward filtering. We also formalise several loss functions that encourage desirable properties on the trajectory paths, and of particular interest are those that directly encourage fewer expected function evaluations. Experimentally, we demonstrate that our model class is capable of capturing the natural dynamics of human behaviour; a notoriously volatile application area. Baseline approaches cannot model this problem.

3. Energy-efficient activity recognition framework using wearable accelerometers Elsts, Atis, Twomey, Niall, McConville, Ryan, and Craddock, Ian Journal of Network and Computer Applications 2020 [Abs] [PDF]

   Acceleration data for activity recognition typically are collected on battery-powered devices, leading to a trade-off between high-accuracy recognition and energy-efficient operation. We investigate this trade-off from a feature selection perspective, and propose an energy-efficient activity recognition framework with two key components: a detailed energy consumption model and a number of feature selection algorithms. We evaluate the model and the algorithms using Random Forest classifiers to quantify the recognition accuracy, and find that the multi-objective Particle Swarm Optimization algorithm achieves the best results for the task. The results show that by selecting appropriate groups of features, energy consumption for computation and data transmission is reduced by an order of magnitude compared with the raw-data approach, and that the framework presents a flexible selection of feature groups that allow the designer to choose an appropriate accuracy-energy trade-off for a specific target application.

4. Detecting Signatures of Early-stage Dementia with Behavioural Models Derived from Sensor Data Poyiadzi, Rafael, Yang, Weisong, Ben-Shlomo, Yoav, Craddock, Ian, Coulthard, Liz, Santos-Rodrı́guez, Raúl, Selwood, James, and Twomey, Niall arXiv preprint arXiv:2007.03615 2020 [Abs] [PDF]

   There is a pressing need to automatically understand the state and progression of chronic neurological diseases such as dementia. The emergence of state-of-the-art sensing platforms offers unprecedented opportunities for indirect and automatic evaluation of disease state through the lens of behavioural monitoring. This paper specifically seeks to characterise behavioural signatures of mild cognitive impairment (MCI) and Alzheimer’s disease (AD) in the \textitearly stages of the disease. We introduce bespoke behavioural models and analyses of key symptoms and deploy these on a novel dataset of longitudinal sensor data from persons with MCI and AD. We present preliminary findings that show the relationship between levels of sleep quality and wandering can be subtly different between patients in the early stages of dementia and healthy cohabiting controls.

5. H4LO: automation platform for efficient RF fingerprinting using SLAM-derived map and poses Kozłowski, Michał, Twomey, Niall, Byrne, Dallan, Pope, James, Santos-Rodrı́guez, Raúl, and Piechocki, Robert J IET Radar, Sonar & Navigation 2020 [Abs] [PDF]

   One of the main shortcomings of Received Signal Strength based indoor localisation techniques is the labour and time cost involved in acquiring labelled ‘ground-truth’ training data. This training data is often obtained through fingerprinting, which involves the user visiting all prescribed locations to capture sensor observations throughout the environment. These prescribed sites must be annotated with reference coordinates which correspond to a known floor plan. In this work, we present ‘H4LO’ (Helmet for Localisation Optimisation): a low-cost robotic system designed to cut down on the labour by utilising an off-the-shelf Light Detection and Ranging device. This system allows for Simultaneous Localisation and Mapping, providing the human user with an accurate pose estimation and a corresponding map of the environment. The high resolution location estimation can then be used to train a positioning model, where Received Signal Strength data is acquired from a human-worn wearable device. The method is evaluated using live measurements, recorded within a residential property in Bristol. We compare the ground-truth location labels generated automatically by the H4LO system with a camera-based fingerprinting technique from previous work. We find that the system remains comparable in performance to the less-efficient camera-based method, whilst removing the need for time-consuming labour associated with registering the user’s location.

2019

1. HyperStream: a Workflow Engine for Streaming Data Diethe, Tom, Kull, Meelis, Twomey, Niall, Sokol, Kacper, Song, Hao, Perello-Nieto, Miquel, Tonkin, Emma, and Flach, Peter arXiv preprint arXiv:1908.02858 2019 [Abs] [PDF]

   This paper describes HyperStream, a large-scale, flexible and robust software package, written in the Python language, for processing streaming data with workflow creation capabilities. HyperStream overcomes the limitations of other computational engines and provides high-level interfaces to execute complex nesting, fusion, and prediction both in online and offline forms in streaming environments. HyperStream is a general purpose tool that is well-suited for the design, development, and deployment of Machine Learning algorithms and predictive models in a wide space of sequential predictive problems. Source code, installation instructions, examples, and documentation can be found at https://github.com/IRC-SPHERE/HyperStream

2. Neural ODEs with stochastic vector field mixtures Twomey, Niall, Kozłowski, Michał, and Santos-Rodrı́guez, Raúl arXiv preprint arXiv:1905.09905 2019 [Abs] [PDF]

   It was recently shown that neural ordinary differential equation models cannot solve fundamental and seemingly straightforward tasks even with high-capacity vector field representations. This paper introduces two other fundamental tasks to the set that baseline methods cannot solve, and proposes mixtures of stochastic vector fields as a model class that is capable of solving these essential problems. Dynamic vector field selection is of critical importance for our model, and our approach is to propagate component uncertainty over the integration interval with a technique based on forward filtering. We also formalise several loss functions that encourage desirable properties on the trajectory paths, and of particular interest are those that directly encourage fewer expected function evaluations. Experimentally, we demonstrate that our model class is capable of capturing the natural dynamics of human behaviour; a notoriously volatile application area. Baseline approaches cannot adequately model this problem.

3. Ordinal Regression as Structured Classification Twomey, Niall, Poyiadzi, Rafael, Mann, Callum, and Santos-Rodrı́guez, Raúl arXiv preprint arXiv:1905.13658 2019 [Abs] [PDF]

   This paper extends the class of ordinal regression models with a structured interpretation of the problem by applying a novel treatment of encoded labels. The net effect of this is to transform the underlying problem from an ordinal regression task to a (structured) classification task which we solve with conditional random fields, thereby achieving a coherent and probabilistic model in which all model parameters are jointly learnt. Importantly, we show that although we have cast ordinal regression to classification, our method still fall within the class of decomposition methods in the ordinal regression ontology. This is an important link since our experience is that many applications of machine learning to healthcare ignores completely the important nature of the label ordering, and hence these approaches should considered naive in this ontology. We also show that our model is flexible both in how it adapts to data manifolds and in terms of the operations that are available for practitioner to execute. Our empirical evaluation demonstrates that the proposed approach overwhelmingly produces superior and often statistically significant results over baseline approaches on forty popular ordinal regression models, and demonstrate that the proposed model significantly out-performs baselines on synthetic and real datasets. Our implementation, together with scripts to reproduce the results of this work, will be available on a public GitHub repository.

4. Active Learning with Label Proportions Poyiadzi, Rafael, Santos-Rodrı́guez, Raul, and Twomey, Niall In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2019 [Abs] [PDF]

   Active Learning (AL) refers to the setting where the learner has the ability to perform queries to an oracle to acquire the true label of an instance or, sometimes, a set of instances. Even though Active Learning has been studied extensively, the setting is usually restricted to assume that the oracle is trustworthy and will provide the actual label. We argue that, while common, this approach can be made more flexible to account for different forms of supervision. In this paper, we propose a new framework that allows the algorithm to request the label for a bag of samples at a time. Although this label will come in the form of proportions of class labels in the bags and therefore encode less information, we demonstrate that we can still learn effectively.

5. An application of hierarchical Gaussian processes to the detection of anomalies in star light curves Twomey, Niall, Chen, Haoyan, Diethe, Tom, and Flach, Peter Neurocomputing 2019 [Abs] [PDF]

   This study is concerned with astronomical time-series called light-curves that represent the brightness of celestial objects over a period of time. We consider the task of finding anomalous light-curves of periodic variable stars. We employ a Hierarchical Gaussian Process to create a general and stable model of time-series for anomaly detection, and apply this approach to the light-curve problem. Hierarchical Gaussian Processes require only a few additional parameters compared to conventional Gaussian Processes and incur negligible additional computational complexity. Moreover, since the additional parameters are objectively optimised in a principled probabilistic framework one does not need to resort to grid searches for parameter selection. Experimentally, we demonstrate that our approach outperforms several baselines on both synthetic and light-curve data. Of particular interest is that the proposed method generalises very well from small subsets of the data, achieving near perfect precision of outlier detection even with as few as seven instances.

2018

1. Talk, Text, Tag? Understanding Self-Annotation of Smart Home Data from a User’s Perspective Tonkin, Emma, Burrows, Alison, Woznowski, Przemysław, Laskowski, Pawel, Yordanova, Kristina, Twomey, Niall, and Craddock, Ian Sensors 2018 [Abs] [PDF]

   Delivering effortless interactions and appropriate interventions through pervasive systems requires making sense of multiple streams of sensor data. This is particularly challenging when these concern people’s natural behaviours in the real world. This paper takes a multidisciplinary perspective of annotation and draws on an exploratory study of 12 people, who were encouraged to use a multi-modal annotation app while living in a prototype smart home. Analysis of the app usage data and of semi-structured interviews with the participants revealed strengths and limitations regarding self-annotation in a naturalistic context. Handing control of the annotation process to research participants enabled them to reason about their own data, while generating accounts that were appropriate and acceptable to them. Self-annotation provided participants an opportunity to reflect on themselves and their routines, but it was also a means to express themselves freely and sometimes even a backchannel to communicate playfully with the researchers. However, self-annotation may not be an effective way to capture accurate start and finish times for activities, or location associated with activity information. This paper offers new insights and recommendations for the design of self-annotation tools for deployment in the real world.

2. Label Propagation for Learning with Label Proportions Poyiadzi, Rafael, Santos-Rodrı́guez, Raul, and Twomey, Niall In 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP) 2018 [Abs] [PDF]

   Learning with Label Proportions (LLP) is the problem of recovering the underlying true labels given a dataset when the data is presented in the form of bags. This paradigm is particularly suitable in contexts where providing individual labels is expensive and label aggregates are more easily obtained. In the healthcare domain, it is a burden for a patient to keep a detailed diary of their daily routines, but often they will be amenable to provide higher level summaries of daily behavior. We present a novel and efficient graph-based algorithm that encourages local smoothness and exploits the global structure of the data, while preserving the ‘mass’ of each bag.

3. A Guide to the SPHERE 100 Homes Study Dataset Elsts, Atis, Burghardt, Tilo, Byrne, Dallan, Camplani, Massimo, Damen, Dima, Fafoutis, Xenofon, Hannuna, Sion, Harwin, William, Holmes, Michael, Janko, Balazs, Ponce Lopez, Victor, Masullo, Alessandro, Mirmehdi, Majid, Oikonomou, George, Piechocki, Robert, Sherratt, Simon, Tonkin, Emma, Twomey, Niall, Vafeas, Antonis, Woznowski, Przemyslaw, and Craddock, Ian arXiv preprint arXiv:1805.11907 2018 [Abs] [PDF]

   The SPHERE project has developed a multi-modal sensor platform for health and behavior monitoring in residential environments. So far, the SPHERE platform has been deployed for data collection in approximately 50 homes for duration up to one year. This technical document describes the format and the expected content of the SPHERE dataset (s) under preparation. It includes a list of some data quality problems (both known to exist in the dataset (s) and potential ones), their workarounds, and other information important to people working with the SPHERE data, software, and hardware. This document does not aim to be an exhaustive descriptor of the SPHERE dataset (s); it also does not aim to discuss or validate the potential scientific uses of the SPHERE data.

4. On-Board Feature Extraction from Acceleration Data for Activity Recognition Elsts, Atis, McConville, Ryan, Fafoutis, Xenofon, Twomey, Niall, Piechocki, Robert, Santos-Rodrı́guez, Raul, and Craddock, Ian In Proceedings of the International Conference on Embedded Wireless Systems and Networks 2018 [Abs] [PDF]

   Modern wearable devices are equipped with increasingly powerful microcontrollers and therefore are increasingly capable of doing computationally heavy operations, such as feature extraction from sensor data. This paper quantifies the time and energy costs required for on-board computation of features on acceleration data, the reduction achieved in subsequent communication load compared with transmission of the raw data, and the impact on daily activity recognition in terms of classification accuracy. The results show that platforms based on modern 32-bit ARM Cortex-M microcontrollers significantly benefit from on-board extraction of time-domain features. On the other hand, efficiency gains from computation of frequency domain features at the moment largely remain out of their reach.

5. Anomaly Detection in Star Light Curves using Hierarchical Gaussian Processes Chen, Haoyan, Diethe, Tom, Twomey, Niall, and Flach, Peter In European Symposium on Artificial Neural Networks (ESANN) 2018 [Abs] [PDF]

   Here we examine astronomical time-series called light-curve data, which represent the brightness of celestial objects over a period of time. We focus specifically on the task of finding anomalies in three sets of light-curves of periodic variable stars. We employ a hierarchical Gaussian process to create a general and stable model of time series for anomaly detection, and apply this approach to the light curve problem. Hierarchical Gaussian processes require only a few additional parameters than Gaussian processes and incur negligible additional computational complexity. Additionally, the additional parameters are objectively optimised in a principled probabilistic framework. Experimentally, our approach outperforms several baselines and highlights several anomalous light curves in the datasets investigated.

6. Person Identification and Discovery With Wrist Worn Accelerometer Data McConville, Ryan, Santos-Rodrı́guez, Raúl, and Twomey, Niall In European Symposium on Artificial Neural Networks (ESANN) 2018 [Abs] [PDF]

   Internet of Things (IoT) devices with embedded accelerometers continue to grow in popularity. These are often attached to individuals, whether they are a mobile phone in a pocket or a smartwatch on a wrist, and are constantly capturing data of a personal nature. In this work we propose a method for person identification using accelerometer data via supervised machine learning techniques. Further, we introduce the first unsupervised method for discovering individuals using the same accelerometer. We report the performance both in terms of classification and clustering using a publicly available dataset covering a large number of activities of daily living. While this has numerous benefits in tasks such as activity recognition and biometrics, this work also motivates the debate and discussion around privacy concerns of the analysis of accelerometer data.

7. Efficient Approximate Representations of Computationally Expensive Features Santos-Rodrı́guez, Raul, and Twomey, Niall In European Symposium on Artificial Neural Networks (ESANN) 2018 [Abs] [PDF]

   High computational complexity is often a barrier to achieving desired representations in resource-constrained settings. This paper introduces a simple and computationally cheap method of approximating complex features. We do so by carefully constraining the architecture of Neural Networks (NNs) and regress from raw data to the intended feature representation. Our analysis focuses on spectral features, and demonstrates how low-capacity networks can capture the end-to-end dynamics of cascaded composite functions. Not only do approximating NNs simplify the analysis pipeline, but our approach produces feature representations up to 20 times more quickly. Excellent feature fidelity is achieved in our experimental analysis with feature approximations, but we also report nearly indistinguishable predictive performance when comparing between exact and approximate representations.

8. A Comprehensive Study of Activity Recognition Using Accelerometers Twomey, Niall, Diethe, Tom, Fafoutis, Xenofon, Elsts, Atis, McConville, Ryan, Flach, Peter, and Craddock, Ian In Informatics 2018 [Abs] [PDF]

   This paper serves as a survey and empirical evaluation of the state-of-the-art in activity recognition methods using accelerometers. The paper is particularly focused on long-term activity recognition in real-world settings. In these environments, data collection is not a trivial matter; thus, there are performance trade-offs between prediction accuracy, which is not the sole system objective, and keeping the maintenance overhead at minimum levels. We examine research that has focused on the selection of activities, the features that are extracted from the accelerometer data, the segmentation of the time-series data, the locations of accelerometers, the selection and configuration trade-offs, the test/retest reliability, and the generalisation performance. Furthermore, we study these questions from an experimental platform and show, somewhat surprisingly, that many disparate experimental configurations yield comparable predictive performance on testing data. Our understanding of these results is that the experimental setup directly and indirectly defines a pathway for context to be delivered to the classifier, and that, in some settings, certain configurations are more optimal than alternatives. We conclude by identifying how the main results of this work can be used in practice, specifically in experimental configurations in challenging experimental conditions.

9. Releasing eHealth Analytics into the Wild: Lessons Learnt from the SPHERE Project Diethe, Tom, Holmes, Mike, Kull, Meelis, Perello-Nieto, Miquel, Sokol, Kacper, Song, Hao, Tonkin, Emma, Twomey, Niall, and Flach, Peter In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining 2018 [Abs] [PDF]

   The SPHERE project is devoted to advancing eHealth in a smart-home context, and supports full-scale sensing and data analysis to enable a generic healthcare service. We describe, from a data-science perspective, our experience of taking the system out of the laboratory into more than thirty homes in Bristol, UK. We describe the infrastructure and processes that had to be developed along the way, describe how we train and deploy Machine Learning systems in this context, and give a realistic appraisal of the state of the deployed systems.

2017

1. SPHERE: A sensor platform for healthcare in a residential environment Woznowski, Przemyslaw, Burrows, Alison, Diethe, Tom, Fafoutis, Xenofon, Hall, Jake, Hannuna, Sion, Camplani, Massimo, Twomey, Niall, Kozłowski, Michal, Tan, Bo, and others, 2017 [Abs]

   It can be tempting to think about smart homes like one thinks about smart cities. On the surface, smart homes and smart cities comprise coherent systems enabled by similar sensing and interactive technologies. It can also be argued that both are broadly underpinned by shared goals of sustainable development, inclusive user engagement and improved service delivery. However, the home possesses unique characteristics that must be considered in order to develop effective smart home systems that are adopted in the real world.

2. Unsupervised learning of sensor topologies for improving activity recognition in smart environments Twomey, Niall, Diethe, Tom, Craddock, Ian, and Flach, Peter Neurocomputing 2017 [Abs] [PDF]

   There has been significant recent interest in sensing systems and ‘smart environments’, with a number of longitudinal studies in this area. Typically the goal of these studies is to develop methods to predict, at any one moment of time, the activity or activities that the resident(s) of the home are engaged in, which may in turn be used for determining normal or abnormal patterns of behaviour (e.g. in a health-care setting). Classification algorithms, such as Conditional Random Field (CRFs), typically consider sensor activations as features but these are often treated as if they were independent, which in general they are not. Our hypothesis is that learning patterns based on combinations of sensors will be more powerful than single sensors alone. The exhaustive approach – to take all possible combinations of sensors and learn classifier weights for each combination – is clearly computationally prohibitive. We show that through the application of signal processing and information-theoretic techniques we can learn about the sensor topology in the home (i.e. learn an adjacency matrix) which enables us to determine the combinations of sensors that will be useful for classification ahead of time. As a result we can achieve classification performance better than that of the exhaustive approach, whilst only incurring a small cost in terms of computational resources. We demonstrate our results on several datasets, showing that our method is robust in terms of variations in the layout and the number of residents in the house. Furthermore, we have incorporated the adjacency matrix into the CRF learning framework and have shown that it can improve performance over multiple baselines.

3. Probabilistic sensor fusion for ambient assisted living Diethe, Tom, Twomey, Niall, Kull, Meelis, Flach, Peter, and Craddock, Ian arXiv preprint arXiv:1702.01209 2017 [Abs] [PDF]

   There is a widely-accepted need to revise current forms of health-care provision, with particular interest in sensing systems in the home. Given a multiple-modality sensor platform with heterogeneous network connectivity, as is under development in the Sensor Platform for HEalthcare in Residential Environment (SPHERE) Interdisciplinary Research Collaboration (IRC), we face specific challenges relating to the fusion of the heterogeneous sensor modalities. We introduce Bayesian models for sensor fusion, which aims to address the challenges of fusion of heterogeneous sensor modalities. Using this approach we are able to identify the modalities that have most utility for each particular activity, and simultaneously identify which features within that activity are most relevant for a given activity.

4. Talk, text or tag? Woznowski, Przemyslaw, Tonkin, Emma, Laskowski, Pawel, Twomey, Niall, Yordanova, Kristina, and Burrows, Alison In 1st International Workshop on Annotation of useR Data for UbiquitOUs Systems at Pervasive Computing’ to PERCOM 2017 2017 [Abs] [PDF]

   Pervasive computing and, specifically, the Internet of Things aspire to deliver smart services and effortless interactions for their users. Achieving this requires making sense of multiple streams of sensor data, which becomes particularly challenging when these concern people’s activities in the real world. In this paper we describe the exploration of different approaches that allow users to self-annotate their activities in near real-time, which in turn can be used as ground-truth to develop algorithms for automated and accurate activity recognition. We offer the lessons we learnt during each design iteration of a smart-phone app and detail how we arrived at our current approach to acquiring ground-truth data ‘in the wild’. In doing so, we uncovered tensions between researchers’ data annotation requirements and users’ interaction requirements, which need equal consideration if an acceptable self-annotation solution is to be achieved. We present an ongoing user study of a hybrid approach, which supports activity logging that is appropriate to different individuals and contexts.

2016

1. The SPHERE challenge: Activity recognition with multimodal sensor data Twomey, Niall, Diethe, Tom, Kull, Meelis, Song, Hao, Camplani, Massimo, Hannuna, Sion, Fafoutis, Xenofon, Zhu, Ni, Woznowski, Pete, Flach, Peter, and others, arXiv preprint arXiv:1603.00797 2016 [Abs] [PDF]

   This paper outlines the Sensor Platform for HEalthcare in Residential Environment (SPHERE) project and details the SPHERE challenge that will take place in conjunction with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery (ECML-PKDD) between March and July 2016. The SPHERE challenge is an activity recognition competition where predictions are made from video, accelerometer and environmental sensors. Monetary prizes will be awarded to the top three entrants, with Euro 1,000 being awarded to the winner, Euro 600 being awarded to the first runner up, and Euro 400 being awarded to the second runner up.

2. Active transfer learning for activity recognition Diethe, Tom, Twomey, Niall, and Flach, Peter In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning 2016 [Abs] [PDF]

   We examine activity recognition from accelerometers, which provides at least two major challenges for machine learning. Firstly, the deployment context is likely to differ from the learning context. Secondly, accurate labelling of training data is time-consuming and error-prone. This calls for a combination of active and transfer learning. We derive a hierarchical Bayesian model that is a natural fit to such problems, and provide empirical validation on synthetic and publicly available datasets. The results show that by combining active and transfer learning, we can achieve faster learning with fewer labels on a target domain than by either alone.

3. On the need for structure modelling in sequence prediction Twomey, Niall, Diethe, Tom, and Flach, Peter Machine Learning 2016 [Abs] [PDF]

   There is no uniform approach in the literature for modelling sequential correlations in sequence classification problems. It is easy to find examples of unstructured models (e.g. logistic regression) where correlations are not taken into account at all, but there are also many examples where the correlations are explicitly incorporated into a—potentially computationally expensive—structured classification model (e.g. conditional random fields). In this paper we lay theoretical and empirical foundations for clarifying the types of problem which necessitate direct modelling of correlations in sequences, and the types of problem where unstructured models that capture sequential aspects solely through features are sufficient. The theoretical work in this paper shows that the rate of decay of auto-correlations within a sequence is related to the excess classification risk that is incurred by ignoring the structural aspect of the data. This is an intuitively appealing result, demonstrating the intimate link between the auto-correlations and excess classification risk. Drawing directly on this theory, we develop well-founded visual analytics tools that can be applied a priori on data sequences and we demonstrate how these tools can guide practitioners in specifying feature representations based on auto-correlation profiles. Empirical analysis is performed on three sequential datasets. With baseline feature templates, structured and unstructured models achieve similar performance, indicating no initial preference for either model. We then apply the visual analytics tools to the datasets, and show that classification performance in all cases is improved over baseline results when our tools are involved in defining feature representations.

4. BDL. NET: Bayesian dictionary learning in Infer. NET Diethe, Tom, Twomey, Niall, and Flach, Peter In 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP) 2016 [Abs] [PDF]

   We introduce and analyse a flexible and efficient implementation of Bayesian dictionary learning for sparse coding. By placing Gaussian-inverse-Gamma hierarchical priors on the coefficients, the model can automatically determine the required sparsity level for good reconstructions, whilst also automatically learning the noise level in the data, obviating the need for heuristic methods for choosing sparsity levels. This model can be solved efficiently using Variational Message Passing (VMP), which we have implemented in the Infer.NET framework for probabilistic programming and inference. We analyse the properties of the model via empirical validation on several accelerometer datasets. We provide source code to replicate all of the experiments in this paper.

2015

1. Bayesian Active Transfer Learning in Smart Homes Diethe, Tom, Twomey, Niall, and Flach, Peter In Advances in Active Learning : Bridging Theory and Practice, ICML 2015 2015 [Abs] [PDF]

   There are at least two major challenges for machine learning in the smart-home setting. Firstly, the deployment context will be very different to the the context in which learning occurs, due to both individual differences in typical activity patterns and different house and sensor layouts. Secondly, accurate labelling of training data is an extremely time-consuming process, and the resulting labels are potentially noisy and error-prone. The resulting framework is therefore a combination of active and transfer learning. We argue that hierarchical Bayesian methods are particularly well suited to problems of this nature, and give a possible formulation of such a model.

2. Bridging e-health and the internet of things: The sphere project Zhu, Ni, Diethe, Tom, Camplani, Massimo, Tao, Lili, Burrows, Alison, Twomey, Niall, Kaleshi, Dritan, Mirmehdi, Majid, Flach, Peter, and Craddock, Ian IEEE Intelligent Systems 2015 [Abs] [PDF]

   There’s a widely known need to revise current forms of healthcare provision. Of particular interest are sensing systems in the home, which have been central to several studies. This article presents an overview of this rapidly growing body of work, as well as the implications for machine learning, with an aim of uncovering the gap between the state of the art and the broad needs of healthcare services in ambient assisted living. Most approaches address specific healthcare concerns, which typically result in solutions that aren’t able to support full-scale sensing and data analysis for a more generic healthcare service, but the approach in this article differs from seamlessly linking multimodel data-collecting infrastructure and data analytics together in an AAL platform. This article also outlines a multimodality sensor platform with heterogeneous network connectivity, which is under development in the sensor platform for healthcare in a residential environment (SPHERE) Interdisciplinary Research Collaboration (IRC).

3. Bayesian Active Learning with Evidence-Based Instance Selection Twomey, Niall, Diethe, Tom, and Flach, Peter In Second International Workshop on Learning over Multiple Contexts in conjunction with ECML PKDD 2015 2015 [Abs] [PDF]

   There are at least two major challenges for machine learning when performing activity recognition in the smart-home setting. Firstly, the deployment context may be very different to the context in which learning occurs, due to both individual differences in typical activity patterns and different house and sensor layouts. Secondly, accurate labelling of training data is an extremely timeconsuming process, and the resulting labels are potentially noisy and error-prone. We propose that these challenges are best solved by combining transfer learning and active learning, and argue that hierarchical Bayesian methods are particularly well suited to problems of this nature. We introduce a new active learning method that is based on on Bayesian model selection, and hence fits more concomitantly with the Bayesian framework than previous decision theoretic approaches, and is able to cope with situations that the simple but naıve method of uncertainty sampling cannot. These initial results are promising and show the applicability of Bayesian model selection for active learning. We provide some experimental results combining two publicly available activity recognition from accelerometry data-sets, where we transfer from one data-set to another before performing active learning. This effectively utilises existing models to new domains where the parameters may be adapted to the new context if required. Here the results demonstrate that transfer learning is effective, and that the proposed evidence-based active selection method can be more effective than baseline methods for the subsequent active learning.

4. Gaussian Process Model Re-Use Diethe, Tom, Twomey, Niall, and Flach, Peter In 2nd International Workshop on Learning over Multiple Contexts (LMCE 2015), in conjunction with ECML PKDD 2015 2015 [Abs] [PDF]

   Consider the situation where we have some pre-trained classification models for bike rental stations (or any other spatially located data). Given a new rental station (deployment context), we imagine that there might be some rental stations that are more similar to this station in terms of the daily usage patterns, whether or not these stations are close by or not. We propose to use a Gaussian Process (GP) to model the relationship between geographic location and the type of the station, as determined by heuristics based on the daily usage patterns. For a deployment station, we then find the closest stations in terms of the Gaussian Process (GP) function output, and then use the models trained on these stations on the deployment station. We compare against several baselines, and show that this method is able to outperform those baselines.

5. Bayesian modelling of the temporal aspects of smart home activity with circular statistics Diethe, Tom, Twomey, Niall, and Flach, Peter In Joint European Conference on Machine Learning and Knowledge Discovery in Databases 2015 [Abs] [PDF]

   Typically, when analysing patterns of activity in a smart home environment, the daily patterns of activity are either ignored completely or summarised into a high-level “hour-of-day” feature that is then combined with sensor activities. However, when summarising the temporal nature of an activity into a coarse feature such as this, not only is information lost after discretisation, but also the strength of the periodicity of the action is ignored. We propose to model the temporal nature of activities using circular statistics, and in particular by performing Bayesian inference with Wrapped Normal \mathcal (WN) and \mathcal WN Mixture \mathcal (WNM) models. We firstly demonstrate the accuracy of inference on toy data using both Gibbs sampling and Expectation Propagation (EP), and then show the results of the inference on publicly available smart-home data. Such models can be useful for analysis or prediction in their own right, or can be readily combined with larger models incorporating multiple modalities of sensor activity.

6. An RSSI-based Wall Prediction Model for Residential Floor Map Construction Fafoutis, Xenofon, Mellios, Evangelos, Twomey, Niall, Diethe, Tom, Hilton, Geoffrey, and Piechocki, Robert In 2nd IEEE World Forum on Internet of Things (WF-IoT), 2015 [Abs] [PDF]

   In residential environments, floor maps, often required by location-based services, cannot be trivially acquired. Researchers have addressed the problem of automatic floor map construction in indoor environments using various modalities, such as inertial sensors, Radio Frequency (RF) fingerprinting and video cameras. Considering that some of these techniques are unavailable or impractical to implement in residential environments, in this paper, we focus on using RF signals to predict the number of walls between a wearable device and an access point. Using both supervised and unsupervised learning techniques on two data sets; a system-level data set of Bluetooth packets, and measurements on the signal attenuation, we construct wall prediction models that yield up to 91% identification rate. As a proof-of-concept, we also use the wall prediction models to infer the floor plan of a smart home deployment in a real residential environment.

2014

1. Context Modulation of Sensor Data Applied to Activity Recognition in Smart Homes Twomey, Niall, and Flach, Peter In Workshop on Learning over Multiple Contexts, European Conference on Machine Learning (ECML’14) 2014 [Abs] [PDF]

   In this paper we present a method of modulating the context of data captured in smart homes. We show that we can dramatically adapt their sensor network topology and that this approach can be used to help understand various aspects of such sensor environments. We demonstrate how, with our software, we can discover the importance of individual sensors, clusters of sensors and sensor categories for resident identification and activity recognition. Finally, we validate the utility of context modulation in a number of experimental scenarios that show how the activity recognition is affected by each sensor topology elicited by these scenarios.

2. SPHERE-a Sensor Platform for HEalthcare in a Residential Environment Diethe, Tom R, Twomey, Niall, and Flach, Peter 2014 [Abs]

   Obesity, depression, stroke, falls, cardiovascular and musculoskeletal disease are some of the biggest health issues and fastest-rising categories of healthcare costs. The associated expenditure is widely regarded as unsustainable and the impact on quality of life is felt by millions of people in the UK each day. The vision of the SPHERE IRC is not to develop fundamentally-new sensors for individual health conditions but rather to impact all these healthcare needs simultaneously through data-fusion and pattern-recognition from a common platform of non-medical/environmental sensors at home. The system will be general-purpose, low-cost and scalable. Sensors will be entirely passive, requiring no action by the user and hence suitable for all patients including the most vulnerable. A central hypothesis is that deviations from a user’s established pattern of behaviour in their own home have particular, unexploited, diagnostic value.

3. A Machine Learning Approach to Objective Cardiac Event Detection Twomey, N, and Flach, P In The 8th International Conference on Complex, Intelligent and Software Intensive Systems 2014 [Abs] [PDF]

   This paper presents an automated framework for the detection of the QRS complex from Electrocardiogram (ECG) signals. We introduce an artefact-tolerant pre-processing algorithm which emphasises a number of characteristics of the ECG that are representative of the QRS complex. With this processed ECG signal we train Logistic Regression and Support Vector Machine classification models. With our approach we obtain over 99.7% detection sensitivity and precision on the MIT-BIH database without using supplementary de-noising or pre-emphasis filters.

2013

1. Twomey, N. (2013). Digital signal processing and artificial intelligence for the automated classification of food allergy. University College Cork. [Abs] [PDF]

   AS a by-product of the ‘information revolution’which is currently unfolding, lifetimes of man (and indeed computer) hours are being allocated for the automated and intelligent interpretation of data. This is particularly true in medical and clinical settings, where research into machine-assisted diagnosis of physiological conditions gains momentum daily. Of the conditions which have been addressed, however, automated classification of allergy has not been investigated, even though the numbers of allergic persons are rising, and undiagnosed allergies are most likely to elicit fatal consequences. On the basis of the observations of allergists who conduct oral food challenges (OFCs), activity-based analyses of allergy tests were performed. Algorithms were investigated and validated by a pilot study which verified that accelerometer-based inquiry of human movements is particularly well-suited for objective appraisal of activity. However, when these analyses were applied to OFCs, accelerometer-based investigations were found to provide very poor separation between allergic and non-allergic persons, and it was concluded that the avenues explored in this thesis are inadequate for the classification of allergy.

2. Real-time allergy detection Gutierrez Rivas, Raquel, Garcia Dominguez, Juan Jesus, Marnane, William P, Twomey, Niall, and Temko, Andriy In Intelligent Signal Processing (WISP), 2013 IEEE 8th International Symposium on 2013 [Abs] [PDF]

   In this paper, we tackle the problem of the food allergic detection in children, based on the analysis of the ECG signal. Through the detection of some changes of this signal, it is possible to detect any reaction before the tested subject experiments any physical reaction or any reaction that could be harmful to his/her life. To be able to realize this process in real-time and with portable devices, it is necessary to reduce the computational cost of the full process, from the ECG analysis to the allergy detection process.

3. Automated detection of perturbed cardiac physiology during oral food allergen challenge in children Twomey, Niall, Temko, Andrey, Hourihane, JO’B, and Marnane, William P IEEE journal of biomedical and health informatics 2013 [Abs] [PDF]

   This paper investigates the fully automated computer-based detection of allergic reaction in oral food challenges using pediatric ECG signals. Nonallergic background is modeled using a mixture of Gaussians during oral food challenges, and the model likelihoods are used to determine whether a subject is allergic to a food type. The system performance is assessed on the dataset of 24 children (15 allergic and 9 nonallergic) totaling 34 h of data. The proposed detector correctly classified all nonallergic subjects (100 % specificity) and 12 allergic subjects (80 % sensitivity) and is capable of detecting allergy on average 17 min earlier than trained clinicians during oral food challenges, the gold standard of allergy diagnosis. Inclusion of the developed allergy classification platform during oral food challenges recorded would result in a 30% reduction of doses administered to allergic subjects. The results of study introduce the possibility to halt challenges earlier which can safely advance the state of clinical art of allergy diagnosis by reducing the overall exposure to the allergens.

4. Monitoring of heart rate variability during oral food challenge could improve patient safety and diagnostic yield: 837 Twomey, N, Temko, A, Cullinane, C, Daly, D, Marnane, WP, and Hourihane, JO Allergy: European Journal of Allergy and Clinical Immunology 2013 [Abs]

   Background: International consensus is that an Oral Food Challenge (OFC) should continue until objective, predetermined clinical stop criteria are met. OFC can cause anaphylaxis. Anaphylaxis appears more common late in OFC, after a large cumulative dose of test allergen, rather than early in OFC after very small doses. Heart rate variability (HRV) is noted before the clinically observable onset of sepsis and seizures in hypoxic ischaemic encephalopathy and other immune mediated disorders. Method: We retrospectively examined HRV, using fully automated computerbased detection of ECG signals, during 24 clinically-indicated open OFC. Baseline HRV was examined before the 1st dose of OFC was given and epochs of HRV during OFC were compared to this baseline, using 18 known HRV factors. OFC was stopped or continued to top dose, according to clinical practice, blind to the HRV data. Results: Fifteen OFC were positive, 9 were negative. HRV was stable in all negative OFC (100% specificity) and HRV changes were noted in 12/15 positive OFC (80% sensitivity). Use of HRV detection during OFC could have led to 70% reduction in cumulative doses administered during positive OFC. Conclusion: Detection of complex features of Heart Rate Variability during oral food challenge can be automated at the bedside. This finding has potential to widen the adoption of diagnostic OFC, by reducing premature termination of OFC due to subjective symptoms and by improving safety in positive OFC by prompting termination of positive OFC at lower doses of test allergen.

2011

1. Allergy detection with statistical modelling of HRV-based non-reaction baseline features Twomey, Niall, Temko, Andrey, Hourihane, Jonathan O’B, and Marnane, William P In Proceedings of the 4th International Symposium on Applied Sciences in Biomedical and Communication Technologies 2011 [Abs] [PDF]

   This paper investigates the automated classification of oral food challenges (’allergy tests’). The electrocardiograms (ECG) of the subjects being tested for allergies were recorded via a wireless mote, and the QRS complexes were manually annotated and 18 features were extracted from the signals. Principal component analysis was used for feature decorelation and dimensionality reduction and diagonal covariance Gaussian mixture models were used to model non-reaction baseline patient condition. The generated subject independent log likelihood plots were used to separate allergic reaction by means of subject adaptive thresholding. The platform resulted in 87% accuracy of classification with 100% specificity. The algorithm presented can detect allergy up to 30 minutes sooner than the current state of the clinical art allergy detection (7minutes ± 9).

2010

1. Comparison of accelerometer-based energy expenditure estimation algorithms Twomey, Niall, Faul, Stephen, and Marnane, William P In 2010 4th International Conference on Pervasive Computing Technologies for Healthcare 2010 [Abs] [PDF]

   Many accelerometer-based energy expenditure estimation algorithms and platforms have been established in recent topical literature, and each boasts a high correlation against the gold standard in energy expenditure measurement, i.e. indirect calorimetry. The aim of this study was to implement a set of these algorithms, run them all over a common dataset and investigate the strengths and weaknesses associated with each. The algorithms were then ported to a SHIMMER device for a real time, mobile and non-invasive energy expenditure estimation solution. High correlations were found between the accelerometer-regressed energy expenditure estimates and the reference dataset both on a computer and SHIMMER-implementation of the algorithms.

2. The effect of lossy ECG compression on QRS and HRV feature extraction Twomey, Niall, Walsh, Noel, Doyle, Orla, McGinley, Brian, Glavin, Martin, Jones, Edward, and Marnane, WP In 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology 2010 [Abs] [PDF]

   This paper describes the performance of beat detection and heart rate variability (HRV) feature extraction on electrocardiogram signals which have been compressed and reconstructed with a lossy compression algorithm. The set partitioning in hierarchical trees (SPIHT) compression algorithm was used with sixteen compression ratios (CR) between 2 and 50 over the records of the MIT/BIH arrhythmia database. Sensitivities and specificities between 99% and 85% were computed for each CR utilised. The extracted HRV features were between 99% and 82% similar to the features extracted from the annotated records. A notable accuracy drop over all features extracted was noted beyond a CR of 30, with falls of 10% accuracy beyond this compression ratio.

3. Classification of biophysical changes during food allergy challenges Twomey, Niall, Faul, Stephen, Daly, Deirdre, Hourihane, Jonathan O’B, and Marnane, William P In 2010 3rd International Symposium on Applied Sciences in Biomedical and Communication Technologies (ISABEL 2010) 2010 [Abs] [PDF]

   This paper details the process of oral food challenges (‘allergy tests’) and steps followed to investigate whether automatic classification of the tests is possible. It has been observed by trained staff that the mood and physiological signals of a subject being tested for allergies can change during the test if they are sensitive to the allergen they are being tested against. Data from thirteen subjects was recorded, and thirteen features were extracted from each of these datasets. The changes in the features were then analysed over the course of each test. It was noted that when a subject failed the challenge, some of the features extracted from the ECG trace changed suddenly near the time that the test was stopped. Threshold classification was employed, and ROC curves were generated. Some features gave rise to ROC areas of over 0.97 on certain subjects. An average ROC area of 0.57 was computed over all subjects and all features due to wide inter subject variability.