data integration platform that can be used by 77 municipalities, which collects data while reducing the burden on municipalities through automated website crawling

Nagano Prefecture DX Promotion Division
Industry/business type
Public/municipal
Products introduced
DataSpider Servista
HULFT DataCatalog
  • data integration platform
  • Data Utilization
  • Crawling
Main image of data integration platform that can be used by 77 municipalities, which enables data collection while reducing the burden on municipalities through automatic website crawling.

Nagano Prefecture has formulated the Nagano Prefecture DX Strategy, which embodies the maximum utilization of cutting-edge technology, as outlined in the Nagano Prefecture Comprehensive Five-Year Plan "Happy Shinshu Creation Plan 2.0," which was formulated in 2018. To realize this strategy, the prefecture has promoted DX in areas such as administrative work and infrastructure through the "Smart Highland Promotion Program," and has also supported DX in all industries within the prefecture through the "Shinshu IT Valley Initiative."

One of the industry-academia collaborative projects undertaken as part of the Smart Highland Promotion Program is the creation of a flood prediction system, and the development of data integration platform is being promoted. DataSpider International being used as the data integration platform, and HULFT DataCatalog is being used as the environment that enables data utilization through visualization.

Customer Issues

Developing data integration platform that enables data collection while reducing the burden on municipalities

Benefits of implementation

Without burdening municipalities
By crawling
Achieve data acquisition

Adding partners and
Item changes, etc.
An environment where in-house production is possible
Maintenance completed

Highly wide-area
data integration platform
in
Hopes for future use

data integration platform required for participating in joint research on background flood forecasting

Director of the DX Promotion Division, Planning and Promotion Department, Nagano Prefecture
Mr. Kiyohiko Nagano

Nagano Prefecture is located almost in the center of Honshu, and is a mountainous prefecture with mountains accounting for 84% of the prefecture's total area. With tourist attractions such as Zenkoji Temple and Matsumoto Castle, popular ski resorts such as Shiga Kogen and Hakuba, and the Northern Alps, known as the "roof of Japan" for its 3,000-meter-high mountains, the prefecture is well-known as a tourist destination for both Japanese and international visitors.

With an eye on the "Society 5.0" era, Nagano Prefecture formulated the "Nagano Prefecture DX Strategy" in 2020 to promote DX throughout the prefecture, making it an attractive region not only for prefectural residents and local businesses, but also for people and businesses from outside the prefecture. According to Nagano Kiyohiko, head of the DX Promotion Division in the Planning and Promotion Department, the DX strategy has been strongly promoted with two driving forces: the "Smart Highland Promotion Program," which promotes DX in the lives of prefectural residents and administration, and the "Shinshu IT Valley Initiative," which promotes DX in prefectural industries.

One of the two driving forces is the "Smart Highland Promotion Program," which focuses on common operations between the prefecture and municipalities, promoting joint use, and is based on the concept of thoroughly reviewing business processes while taking into consideration the use of cloud services and the scalability of IT systems. "As we move forward with seven projects, including the Smart Municipality Promotion Project, Nagano Prefecture decided to participate in a joint research project led by the University of Tokyo and JAXA on improving the accuracy of flood forecasts by utilizing data such as water levels. At that time, we needed a system to effectively collect river data, water levels, and rainfall data held by Nagano Prefecture and municipalities," explains Danbara Masaki of the same department.

data integration platform that supports crawling functions that do not place a burden on the implementing municipalities

Nagano Prefecture has the second-largest number of cities, towns, and villages in Japan, so collaboration with each local government was crucial to creating a new environment. "While we don't have any large cities designated as government ordinance cities, we have 77 cities, towns, and villages, including smaller ones. Working together to promote digital transformation is one of the features of the digital transformation we're promoting in Nagano Prefecture. We needed data integration platform to effectively collect the data these municipalities possess," Nagano explains. While many digitalization initiatives had been underway in individual fields, an environment for data integration across multiple fields hadn't yet been established. "We thought this joint research on flood prediction would be a good opportunity to establish a new data integration platform. Recently, the national government has been discussing the importance of data integration platform for data utilization, so we planned to establish a new platform using the Ministry of Internal Affairs and Communications' Smart City Promotion Project Subsidy for Resolving Regional Issues," Nagano recalls.

Therefore, they determined the necessary requirements for data integration platform, focusing on the Ministry of Internal Affairs and Communications' requirements for subsidized projects, such as the use of a cloud environment, high scalability, and efforts to reduce project costs for the development of a sustainable environment. One of the functional requirements unique to the prefecture was the implementation of a crawling function to acquire information directly from the website. In general, data is retrieved using an API, but the municipalities have not developed an environment that takes the connection to data integration platform into account. Therefore, it was anticipated that individual modification costs would be incurred. data integration platform Since many of the municipalities had no experience in data utilization using APIs, it was necessary to start data collection in a way that would not place a burden on the municipalities by crawling their Web sites and acquiring the data on the prefectural side," said Nagano. In order for them to recognize the usefulness of data utilization, it was first important to implement a function that would allow them to collect data without any burden.

Additionally, in order to reduce project costs, one of the requirements was that the tool be a low-code tool that could be developed in-house as data integration platform. In fact, Nagano Prefecture has been actively recruiting digital talent by establishing digital-specific job categories from fiscal year 2022, and the DX Promotion Division has been recruiting talent with IT experience, such as Mr. Danbara and division chief Mr. Yoshizawa Yukinobu. "We selected the platform with the aim of creating an environment where data additions and changes could be done in-house without outsourcing, including for our own staff," says Nagano.

Chief of DX Promotion Division, Planning and Promotion Department, Nagano Prefecture
Mr. Masaki Danbara

To procure a new data integration platform, a bidding process was held using a proposal system, and ultimately data integration platform construction service provided by Media Force Co., Ltd. (headquarters: Shinjuku-ku, Tokyo), a partner of SAISON INFORMATION SYSTEMS CO., LTD. Co., Ltd. (now Saison Technology), was selected, with DataSpider International being introduced as the core of data integration platform, and HULFT DataCatalog as the visualization tool to promote data utilization. "Of course, we evaluated the overall proposal highly, but there's no doubt that DataSpider International, which has a high market share in data integration, including globally, gave us a sense of security and trust in terms of support," says Danbara.

Effect: Successfully developed data integration platform that can be used by all local governments in the prefecture

Chief, DX Promotion Division, Planning and Promotion Department, Nagano Prefecture
Mr. Yukinobu Yoshizawa

The Nagano Prefecture Smart Highland data integration platform Construction Project, which was recently implemented, built data integration platform centered on DataSpider International, and visualized the information by automatically collecting and cataloging the metadata of the information accumulated in HULFT DataCatalog. Operation began in April 2023, and information such as water levels, rainfall, and snowfall is regularly collected by crawling from 10 sites managed by the prefecture and 8 websites of municipalities.

"Since the update frequency varies from website to website, we crawl and update information at intervals ranging from every 10 minutes at the shortest to every hour at the longest," explains Yoshizawa. The plan is to continue selecting websites to access and increase the number of connections with the understanding of local governments.

The collected data can be used not only for joint research, but also by companies that wish to use it by submitting an electronic application to the prefecture, and data has already been provided to several companies. The DX Promotion Division is also using HULFT DataCatalog as a dashboard to visualize information. "We have secured enough accounts to link all 77 cities, towns, and villages in the prefecture, but we are currently starting small, and are now in the phase of considering how we can use it to create value," says Nagano. In the future, they would like to consider ways to provide data that will enable early evacuation of prefectural residents in the event of a disaster, and to use the data in disaster prevention.

data integration platform image

The most significant benefit of using DataSpider International and HULFT DataCatalog to develop data integration platform has been the smooth collection of various data needed for flood predictions, without placing a burden on municipalities. "Thanks to our unprecedented crawling approach, we have succeeded in significantly reducing the burden on municipalities. Also, when handling personal data, it can be difficult to get started because of the high hurdle of opting in, which requires municipalities to provide consent. This time, we limited the use to open data, which significantly lowered the barrier to entry for municipalities. By making people realize the usefulness of data utilization, we have also been able to use this platform development as an opportunity to create an environment that makes it easier to accelerate efforts to develop data integration platform," says Danbara.

What we really appreciate about DataSpider International is that it allows you to easily add processes without writing code.

"I was originally a programmer myself, but even people with little knowledge can easily get involved. It's so much easier than developing with code. Because we can create it ourselves by hand, it also helps to improve the skills of our staff," said Danbara. Yoshizawa praised HULFT DataCatalog, saying, "It has a very easy-to-read, intuitive interface, so even staff who are not digital specialists should be able to use it."

Outlook: We want to develop this into data integration platform that can be used throughout the prefecture

Currently data integration platform is being developed for joint research into flood prediction, but as not all 77 municipalities are participating, they plan to continue active discussions and exchange opinions with each other in order to develop data integration platform that can be used throughout the prefecture. Specifically, it is entirely conceivable that the collection of wide-area data could be expanded to applications that can create new added value in areas other than flooding, such as disaster prevention, welfare, and transportation.

"Rather than using it for administrative purposes, we will likely be exploring ways to collect and use data that can be utilized in the lives of prefectural residents. We would like to steadily build up efforts to collect and utilize data through individual projects like this one that will lead to the resolution of local issues," Nagano said, expressing his hopes. Going forward, rather than collecting and utilizing data individually within each municipality, he would like to create opportunities for dialogue with partner municipalities and deepen discussions about new ways to expand the use of data, so that it can be used to expand data utilization over a wider area. He is enthusiastic about continuing to actively utilize data integration platform, seeing it as a starting point for promoting DX throughout the prefecture.

Furthermore, with the number of employees expected to decrease in both Nagano Prefecture and each city, town, and village, it is entirely conceivable that unless overall optimization is achieved by expanding public services, sustainability will no longer be guaranteed.

In order to maintain the quality of resident services and further improve it, it is necessary to effectively utilize a highly wide-area system such as data integration platform. "We expect the direction of data integration and data utilization promoted by the Digital Agency to continue, and the importance of data integration platform we have now established will likely increase. In line with these government trends, we would like to respond appropriately by thoroughly training personnel who can handle data integration platform," said Nagano, speaking about the future.

Nagano Prefecture DX Promotion Division

  • location
    692-2 Minaminaganohabashita, Nagano City, Nagano Prefecture
  • overview
    He is responsible for promoting DX in Nagano Prefecture. He promotes DX in areas such as the daily lives of citizens and industry using digital technology and data, as well as DX in administrative affairs in cooperation with municipalities. He also works to develop and attract digital talent and develop communication infrastructure.
  • The content of this case study is current as of the time of the interview. The content of this case study may change without notice.
See all support cases