Data Modeling
Mathematical and Computational Data Modeling
0
Citations
4.5k
Views
29
Articles
Volume 1 Number 1 2020
RESEARCH ARTICLE (Open Access)
Federated Learning-Based Movie Genre Prediction from Multimodal Content Using Deep Learning: A Privacy-Preserving Approach
Zulkarnain Saurav1*
Data Modeling 1 (1) 1-9 https://doi.org/10.25163/data.1110839
Submitted: 25 October 2020 Revised: 17 December 2020 Accepted: 23 December 2020 Published: 25 December 2020
Abstract
Predicting a movie's genre well is deceptively hard, and most existing approaches, arguably, have not tried hard enough — they tend to lean on a single data source, either visual scene cues or scraped plot text, which leaves a fair amount of a film's identity unaccounted for. Combining posters and storylines seems like the obvious fix, except that pooling multimodal data this way raises a problem few prior studies have taken seriously: doing so typically requires centralizing sensitive content, an approach that sits uneasily with growing concerns around data breaches, poster piracy, and user distrust of centralized systems. This study set out to address both gaps at once. We collected plot summaries and poster images from IMDB, then built a hybrid federated learning (FL) architecture in which text and image data were processed independently — BERT paired with a one-dimensional CNN for multi-label text classification, and VGG-16 for extracting genre-relevant visual features — before their outputs were fused through a late cross-modality representation and aggregated centrally using FedAvg, ensuring that raw data never left its originating client. The resulting model achieved 80% accuracy, a figure that compares favorably to prior unimodal and non-privacy-preserving genre-prediction studies, while never requiring poster or text data to be shared outright. Taken together, these results suggest that multimodal fusion and federated privacy preservation need not come at each other's expense; if anything, this pipeline indicates the two can be pursued jointly, without meaningfully sacrificing predictive performance, in a way that earlier single-modality or privacy-agnostic approaches simply did not attempt.Keywords: Federated Learning; Multimodal Data; Movie Genre Prediction; Deep Learning; Data Privacy
References
Agbley, B. L. Y., Li, J., Haq, A. U., Bankas, E. K., Ahmad, S., Agyemang, I. O., Kulevome, D., Ndiaye, W. D., Cobbinah, B., & Latipova, S. (2021). Multimodal melanoma detection with federated learning. In 2021 18th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) (pp. 238–244). IEEE.
Arjevani, Y., & Shamir, O. (2015). Communication complexity of distributed convex learning and optimization. In Advances in Neural Information Processing Systems (Vol. 28, pp. 1756–1764).
Chen, W., Ma, M., Ye, Y., Zheng, Z., & Zhou, Y. (2018). IoT service based on JointCloud blockchain: The case study of smart traveling. In Proceedings of the IEEE Symposium on Service-Oriented System Engineering (SOSE) (pp. 216–221).
Fu, X., Wang, H., Shi, P., Fu, Y., & Wang, Y. (2017). JCLedger: A blockchain-based distributed ledger for JointCloud computing. In Proceedings of the IEEE 37th International Conference on Distributed Computing Systems Workshops (ICDCSW) (pp. 289–293).
Hassine, N. B., Milocco, R., & Minet, P. (2017). ARMA based popularity prediction for caching in content delivery networks. In Proceedings of the 2017 Wireless Days (pp. 113–120).
Khan, E. M., Mukta, M. S. H., Ali, M. E., & Mahmud, J. (2020). Predicting users’ movie preference and rating behavior from personality and values. ACM Transactions on Interactive Intelligent Systems, 10(3), 1–25.
Liu, W., Zhang, J., Liang, Z., Peng, L., & Cai, J. (2018). Content popularity prediction and caching for ICN: A deep learning approach with SDN. IEEE Access, 6, 5075–5089.
Liu, Y., Yu, F. R., Li, X., Ji, H., & Leung, V. C. M. (2018). Distributed resource allocation and computation offloading in fog and cloud networks with non-orthogonal multiple access. IEEE Transactions on Vehicular Technology, 67(12), 12137–12151.
Liu, Y., Zhi, T., Xi, H., Duan, X., & Zhang, H. (2019). A novel content popularity prediction algorithm based on auto regressive model in information-centric IoT. IEEE Access, 7, 27555–27564.
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & y Arcas, B. A. (2016). Communication-efficient learning of deep networks from decentralized data (arXiv Preprint arXiv:1602.05629). https://arxiv.org/abs/1602.05629
Mukta, M. S. H., Khan, E. M., Ali, M. E., & Mahmud, J. (2017). Predicting movie genre preferences from personality and values of social media users. In Proceedings of the International AAAI Conference on Web and Social Media (Vol. 11, pp. 624–627).
Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 1532–1543).
Povey, D., Zhang, X., & Khudanpur, S. (2014). Parallel training of deep neural networks with natural gradient and parameter averaging (arXiv Preprint arXiv:1410.7455). https://arxiv.org/abs/1410.7455
Saumya, S., Kumar, J., & Singh, J. P. (2018). Genre fraction detection of a movie using text mining. In Advanced Computing and Systems for Security (Vol. 5, pp. 167–177).
Tang, D., Qin, B., Yang, Y., & Liu, T. (2015). User modeling with neural network for review rating prediction. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI) (pp. 1340–1346).
Wang, S., Wang, T., Sidiropoulos, T., Kantarci, B., He, T., & Poor, H. V. (2018). When edge meets learning: Adaptive control for resource-constrained distributed machine learning. In Proceedings of the IEEE Conference on Computer Communications (INFOCOM) (pp. 63–71).
Wang, X., Chen, M., Taleb, T., Ksentini, A., & Leung, V. C. M. (2014). Cache in the air: Exploiting content caching and delivery techniques for 5G systems. IEEE Communications Magazine, 52(2), 131–139.
Wang, Z., Zhang, Y., Chen, H., Li, Z., & Xia, F. (2018). Deep user modeling for content-based event recommendation in event-based social networks. In Proceedings of the IEEE Conference on Computer Communications (INFOCOM) (pp. 1304–1312).
Xiong, B., Yang, X., Qi, F., & Xu, C. (2020). A unified framework for multi-modal federated learning. Neurocomputing, 480, 110–118.
Yue, X., Wang, H., Liu, W., Li, W., Shi, P., & Ouyang, X. (2018). JCDTA: The data trading architecture design in joint cloud computing. In Proceedings of the IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS) (pp. 1–6).
Zhou, H., Hermans, T., Karandikar, A. V., & Rehg, J. M. (2010). Movie genre classification via scene categorization. In Proceedings of the 18th ACM International Conference on Multimedia (pp. 747–750).
Recommended articles
Comprehensive Review of Foundation Toxicity Models Integrating In Vivo, In Vitro, and Chemical Knowledge for Unified Risk Prediction
1
Save
Save
0
Citation
Citation
21
View
View
0
Share
Share