About Zentsuji City, Kagawa Prefecture
Area: 39.93 km²
Population: 30,448 (as of October 1, Reiwa 5 [2023])
Zentsuji City in Kagawa Prefecture is flanked by Mount Ōsa to the south and Gogaku Mountain to the west, with the Sanuki Plain extending to the east and north. Thanks to its Seto Inland Sea climate, the region is warm, blessed with resources, and has fewer disasters, making it a comfortable place to live. The urban area is centered around the main temple of Zentsuji, forming a unique configuration with many public institutions located in the city center.
For this case study, we had the opportunity to speak with Yasuhiko Okita, Section chief of the Digital Promotion Division of the General Affairs Department in Zentsuji City, about the background, effects, and his impressions of the pilot test of 'Crew,' a tool enabling safe use of ChatGPT within the city office.
Question:Thank you for your cooperation.. To begin, I would like to extend our gratitudefor the visit our team members made to the Zentsuji City Hall the other day.
Answer:Thank you as well for coming. In Zentsuji City, we announced the start of thepilot test for Crew during a press conference. We received various questions atthat time, and as a result, an interview with NHK was arranged. There's astrong industry-wide interest in understanding how generative AI can be used inlocal governments.
Youaccompanied us during that interview and witnessed our staff actually usingCrew in their work.
Reference Link: https://www3.nhk.or.jp/lnews/takamatsu/20230830/8030016744.html
Question:As you mentioned, having the opportunity to see Crew in action really helped usvisualize its use more clearly. Could you please explain once again thebackground of the Crew pilot test?
Answer:In our city, we were hopeful that generative AI services, including ChatGPT,would lead to increased efficiency in our operations. We recognized these asdigital technologies and services that should be utilized as needed. At thesame time, we understood these technologies and services not only as tools toimprove information security awareness but also as a means to challenge andenhance the initiative of our staff in their work. Therefore, understanding thefeatures of these services was deemed necessary for their use. While we wereevaluating various generative AI services, we heard about Crew from your salesteam and became interested.
Question:Among various ChatGPT services availablefor local governments, can you tell us why you decided on Crew?
Answer: In conducting proof-of-concept experiments forbusiness applications, we focused on a feature less common in existing services- the ability to generate responses from documents - and thus, we adopted'Crew.' Crew also boasts a comprehensive admin panel. For instance, we found itconvenient that if personal information is accidentally sent, it can be viewedby the administrators through logs. Moreover, in such cases, even if personalinformation is mistakenly sent, Crew detects and blocks this information, so itis visible to the admin side but not transmitted to OpenAI. It was also apositive point that Crew is already implemented in local governments andfinancial institutions, ensuring that it has all the essential securityfeatures.
Question:How did you utilize Crew?"
Answer:"In this pilot test, our goal was to examine the convenience of using ageneration service with documents within our organization. Therefore, withCrew, we primarily used it for creating new document drafts referencinginternal administrative documents, and for generating Q&As based on ourcity's unique regulations and ordinances. As generative AI has become a topicof discussion, we wanted to test its practical applicability in our jobs. Wehad previously discussed this with the relevant departments and issued accountsto nine divisions that expressed an interest in using it. These divisions arethe Secretary and Public Relations Division, Policy Division, General AffairsDivision, Digital Promotion Division, Accounting Division, Tax Division, SocialWelfare Division, Business Division, and Education General AffairsDivision."
Question:"I understand you've attempted to generate responses based on the city'sunique data. What kind of documents did you upload and use for thispurpose?"
Answer:"We had a revision of our security policy within the agency just thisJuly. In light of this, there was a need to explain the security policy to ourstaff in a clear and understandable manner. So, we decided to upload the city'ssecurity policy document and used it to create a guidance document for ourstaff."
Question:What kind of question were you asked?
Answer: I asked Crew to"Summarize the basic policy of Zentsūji City's security policy in aneasy-to-understand manner." Crew provided an output that captured the keypoints: "The basic policy of Zentsūji City's security policy was formulatedin July of Reiwa 5 (2023). It is reviewed as necessary based on the results ofinformation security audits and self-inspections. Moreover, ..." Crew'sresponse was accurate even from a human perspective, so I was able to use itdirectly to inform our staff. Asking questions to Crew allows us to efficientlypick up the key points, which I find particularly effective in the process ofstaff creating documents.
Question: "What has been theimpact since implementing Crew?"
Answer: "Take the example ofthe security policy I mentioned earlier. Manually reading documents andformulating responses used to take about 4 hours. With Crew, however, itquickly summarizes key points, so the whole process, including staff review ofthe text, now takes about an hour. This is especially beneficial for staffmembers who are not adept at creating documents, as Crew compensates for this.This task was a sporadic one, not part of our routine work, and it involvedreviewing a document we hadn't seen in a while. Particularly in such cases, Ifound the use of generative AI tools like Crew to be very effective."
Question: "It's a significantimprovement in operational efficiency, isn't it? What other kinds of documentshave you uploaded?"
Answer: "For example, weuploaded the Digital Transformation (DX) promotion plans of othermunicipalities. By analyzing these documents with Crew, we could referenceideas that might be applicable to our city's plan. One of the documents was adetailed 40-page report, and it was challenging to determine which elementswere relevant to our city. If a person were to search through the document forapplicable sections one by one, it would have been quite a task.
However, by attempting to generateanswers with Crew, we were able to create a draft. Then, our staff fleshed itout, completing the task in about an hour. Without Crew, it would have likelytaken 3 to 4 hours, so we managed to reduce our workload to about a quarter ofwhat it would have been traditionally."
Question: Thank you. What kind ofquestion did you ask Crew at that time?
Answer: We asked Crew,"Please create a draft plan for DX (Digital Transformation) promotion forZentsuji City, referencing the DX promotion plan of ○○ Town." In response,Crew extracted elements that could serve as references for othermunicipalities, such as "The draft plan for Zentsuji City's DX promotionmay include promoting competition between the public and private sectors,digitizing information, ICT integration in operations, etc." Based onthis, our personnel were able to add appropriate wording to create thedocument. Normally, such a task would take a significant amount of time forstaff members to start from scratch, but with Crew, we were able to get thedesired answers quickly and easily build upon the foundation that was alreadylaid out.
Question: "This time, Section9 and many other sections have implemented the tool. How was it used in othersections?"
Answer: "It was used toupload documents related to the Electronic Bookkeeping Law and to confirm thecontents with the personnel in the accounting section. The way it is usedvaries depending on how questions are asked, but there was a sense of securityin knowing that it could reliably extract all the important provisions withoutomission. For personnel in the accounting section, who were dealing with thelaw for the first time like in this case, it was beneficial that Crew couldsummarize the documents and pinpoint the relevant parts."
Question:"It's great that thissupports those who are new to their jobs. Until now, we have heard about usecases in documents, but how was it to have a conversation through text only?"
Answer (In the Context of Crew):
"In Crew, we have bothdocument and text channels, and we tried both. In terms of user experience , Ithought the responses and the content of the answers in text conversations werecomparable to ChatGPT. For the text channel, for example, we created thank-younotes for respondents of a city event survey and notification letters forwinners of thank-you gifts. When we spoke to the staff who tried this, theyresponded that it was quite useful for their actual work and could be effectively utilized."
Question: "Having actuallyused it, how was the overall reaction of the staff?"
Answer: "Based on the surveywe received from Crafters, we conducted a survey among the staff. As a result,in response to the question 'Do you think Crew can be utilized in your work?',60% of the staff responded that it could be effectively utilized. While somestaff expressed their expectations, it's significant to note that some councilmembers have been using ChatGPT to prepare their responses, indicating theimportance the staff place on mastering such tools. "
Question: We've been hearing aboutsuccessful cases, but we would like to know if there were any instances wherethings didn't work out as expected.
Answer: While the feature thatdetects personal information is useful, there have been occasions where itresponded to messages that weren't personal information, such as public figures(e.g., mayors or council members). In such cases, we told our staff that theycould send the message as is without Crew's intervention. It would be even moreconvenient if Crew could become more familiar with Japanese information.
Furthermore, we felt that theaccuracy of responses varied depending on the type of document. I was able toobtain good answers on my end, but some staff members encountered sentenceswith specific terminology unique to local government documents. If Crew couldachieve a level of accuracy that understands such local government-specificphrasing, it would be even more versatile for a wide range of tasks."
Q: Do you have any suggestions forimprovement? Are there any improvement suggestions for the future?
Answer: Administrative staff oftenperform their tasks by referring to regulations and rules, so we expect animprovement in the accuracy of responses, especially for complex rule sets likeordinance collections. Deciphering and summarizing such content is challengingeven for humans, but we believe that as AI's accuracy improves, it will findmore applications within the agency. Additionally, besides what I mentionedearlier, we attempted to use Crew for various documents. However, for fileswith many diagrams and tables, there is a need for pre-conversion into text, soif Crew can recognize and generate responses directly from diagrams and tables,it would become even more convenient as there would be minimal need forpre-processing.
We aim to further improve ourcapabilities to generate accurate responses even from complex texts anddiagrams. Thank you very much for your time!